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ABSTRACT 


Decisions  within  Complex  Systems:  An  Experimental 
Approach  Using  the  STRATEGEM-2  Computer  Game 

by  J.  Robert  Bois 

In  1989,  John  Sterman  published  his  seminal  paper.  Misperceptions  of  Feedback  in 
Dynamic  Decision  Making.  His  misperception  of  feedback  hypothesis  deals  with  the 
difficulty  people  have  in  managing  complex  environments  even  when  they  purportedly 
have  perfect  knowledge  and  have  perfect  information  about  the  system.  Over  the  years, 
several  authors  have  attempted  to  consider  how  the  human  failures,  which  are  a  prominent 
part  of  the  misperception  of  feedback  hypothesis,  can  be  reduced.  However,  these  authors 
have  achieved  mixed  results  in  attempting  to  make  improvements  to  human  decision 
support.  It  is  the  purpose  of  the  current  research  to  provide  meaningful  decision  support  to 
managers  of  complex  environments.  Specifically,  the  research  used  the  STRATEGEM-2 
simulation  game  and  purposely  developed  a  decision  support  method  designed  to  improve 
human  performance  within  a  complex  system.  The  experiment  required  subjects  to  make  a 
single  decision  within  a  dynamic  system  where  the  task  involved  feedback  delays, 
nonlinearity  of  system  processes,  positive  feedback  loops,  and  multiple  cues.  The  decision 
support  included  a  decision  rule  and  a  newly  developed  game  instruction  designed  to 
improve  participant  knowledge  and  information  about  the  microeconomy  of  the 
STRATEGEM-2  simulation.  Results  of  the  research  have  discovered  that  the  new 
instruction  and  the  decision  support  rule  produced  significant  results  in  improving  decision 
making.  Additionally,  this  research  demonstrates  that  the  lack  of  participant  motivation 
levels  can  mask  decision  support  interventions.  Subjects  with  high  self-assessed  motivation 
outperformed  those  subjects  with  lesser  motivation  levels. 
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Chapter  1 


INTRODUCTION 

Over  nearly  two  decades,  John  Sterman  has  researched  and  written  about  dynamic 
decision-making  of  participants  using  the  STRATEGEM-2  computer  simulation  game.  In 
a  seminal  work  (1989a),  along  with  several  other  accompanying  articles  (1985,  1987, 
1989b,  1994),  he  established  that  a  misperception  of  feedback  in  decision  environments 
exists  on  behalf  of  participants  because  they  fail  to  take  into  account  delays  between  their 
own  decisions  and  the  dynamics  of  the  simulation  environment.  Further,  Sterman  suggests 
that  participants  are  operating  with  perfect  knowledge  of  the  system  structure  along  with 
perfect  information.  However,  given  these  “perfect”  settings,  participants  consistently 
perform  poorly.  Sterman  (1989a)  developed  a  misperception  of  feedback  hypothesis 
misperception  of  feedback  hypothesis  from  his  research.  See  Chapter  2,  The 
STRATEGEM-2  Game,  for  an  expanded  description  of  the  misperception  of  feedback 
hypothesis. 

George  Richardson  and  John  Rohrbaugh  (1990)  essentially  challenged  Sterman’ s 
(1987,  1989a)  findings  by  hypothesizing  that  if  participants  were  given  better  cues  to 
consider  in  the  simulation  environment  they  would  perform  better.  They  replicated  the 
Sterman  (1987,  1989a)  study  using  a  changed  interface  that  incorporated  revised  cue 
designs.  The  results,  unfortunately  were  not  as  predicted.  They  were  mbced  -  one  half  of 
the  participants  improved  their  scores  while  the  other  half  performed  worse. 
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Edward  Howie,  Sharleen  Sy,  Louisa  Ford,  and  Kim  Vicente  (2000),  revisited 
Sterman’s  (1989a)  misperception  of  feedback  hypothesis  and  once  again  attempted  to 
improve  upon  poor  participant  performance.  Howie  and  others’  (2000)  approach  was  very 
similar  to  Richardson  and  Rohrbaugh  (1990)  with  respect  to  how  the  simulation 
information  was  presented  to  the  participant.  Additionally,  they  expanded  their  focus  to 
include  measuring  the  level  of  environment  knowledge  possessed  by  each  participant.  This 
was  an  important  step  forward,  in  that  the  assumptions  made  by  “perfect  knowledge”  had 
not  been  tested  up  to  this  point.  Unfortunately,  like  the  Richardson  and  Rohrbaugh  (1990) 
experiment,  Howie  and  others  (2000)  achieved  mixed  results.  However,  they  did  conclude 
that  improving  how  information  is  presented  to  game  players  does  result  in  improved  game 
scores. 

Problem  Statement 

It  is  possible  that  the  above  findings  have  not  completely  resolved  the  issues 
surrounding  the  misperceptions  of  feedback  hypothesis.  A  major  concern  is  that  the 
misperception  of  feedback  hypothesis  puts  undue  emphasis  upon  the  notion  that 
participants  have  “perfect  knowledge  of  the  system  stmcture  along  with  perfect 
information.”  It  is  more  than  likely  that  this  cannot  be  so,  and  it  was  demonstrated  to  some 
degree  by  Richardson  and  Rohrbaugh  (1990)  and  Howie  and  others  (2000).  For  example, 
Howie  and  others  (2000)  demonstrated  that  knowledge  of  the  system  structure  was  far 
fi’om  ideal  before  (and  even  after)  the  experiment  had  taken  place.  The  Howie  and  others 
(2000)  study,  along  with  Richardson  and  Rohrbaugh  (1990)  also  made  valid  criticisms  on 
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how  the  participants  of  the  Sterman  (1987,  1989a)  studies  did  not  actually  have  perfect 
information  at  their  disposal. 

Therefore,  the  problem  statement  for  the  current  research  is:  That  the  explanations  of 
poor  performance  produced  in  the  Sterman  (1987, 1989a)  studies  may  have  been  flawed,  at 
least  to  the  extent  of  the  “perfect  knowledge  and  perfect  information”  line  of  reason.  It  may 
be  possible,  then,  to  train  or  aide  participants  to  be  better  performers. 

Why  is  this  Important? 

The  problem  statement  is  important  for  the  following  reasons: 

First,  Richardson  and  Rohrbaugh  (1990)  point  out  that  the  information  presented  in  the 
Sterman  (1987,  1989a)  studies  is  less  than  adequate.  Specifically,  they  determined  that  in 
order  for  participants  to  make  the  most  out  of  the  information  presented  by  the  Sterman 
(1987,  1989a)  simulation,  they  would  require  a  certain  degree  of  sophistication  that  most 
likely  would  not  reside  with  the  average  player.  In  other  words,  information  can  be  better 
presented,  along  with  assistance  for  cue  interpretation  that  should  result  in  improved 
participant  performance. 

Second,  Howie  and  others  (2000)  produce  a  convincing  argument  that  Sterman  (1989a) 
did  not  provide  appropriate,  or  adequate,  information  on  the  computer  display  of  his 
simulation.  They  suggest  that  substandard  performance  on  behalf  of  participants  is  not  due 
to  a  lack  of  knowledge  or  to  psychological  limitation. 
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Third,  Howie  and  others  (2000)  demonstrated  that  the  premise  of  “perfect  knowledge” 
does  not  exist.  Participants  who  were  tested  a  priori  and  a  posteriori  exhibited  knowledge 
that  was  far  less  than  optimal. 

Fourth,  when  preparing  for  this  undertaking,  Rohrbaugh^  suggests  that  it  has  become 
evident  that  the  “setup”  of  the  experiment  is  equally  crucial  to  the  actual  experiment  itself 
Apparently,  and  too  often,  researchers  provide  participants  with  instructions,  and  then, 
“jump”  right  into  the  data  collection  process.  In  other  words,  not  enough  attention  has  been 
paid  to  how  participants  are  instructed.  Is  it  possible  then,  that  participants  can  be  better 
prepared,  or  informed,  regarding  the  dynamics  of  the  simulation  environment  that  they  are 
about  to  take  part?  Possibly  so. 

Fifth,  and  most  importantly,  it  is  imperative  to  learn  how  to  improve  dynamic  decision¬ 
making  support.  If  indeed  participants  in  an  experimental  setting  can  learn  how  to  improve 
their  performance  with  simulated  complexity,  then  it  may  be  possible  to  design  decision 
support  systems  to  assist  real  decision  makers  with  the  complexities  they  face  in  real 
systems. 

Finally,  one  should  not  impart  from  this  research  that  it  is  an  attempt  to  debunk  the 
misperception  of  feedback  hypothesis.  To  the  contrary,  although  the  misperception  of 
feedback  hypothesis  may  be  based,  in  part,  on  an  incorrect  assmnption  (that  participants 
have  perfect  knowledge  and  information),  it  remains  important  to  realize  that  human 
judges  have  difficulty  with  delayed  feedback  systems.  Therefore,  it  is  equally  important  to 
explore  methods  that  can  be  used  to  improve  human  performance. 


^  2000.  Rohrbaugh,  J.  Personal  interview.  4  January. 
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Purpose 


In  light  of  the  above  citations,  it  is  the  opinion  of  this  researcher  that  it  is  warranted  to 
try  “once  again”  and  see  if  participants  can  indeed  perform  better.  There  are  too  many 
“mixed  results”  requiring  further/additional  exploration. 


Research  Hypotheses 

Assuming  there  are  ways  to  improve  human  performance  in  the  face  of  time-delayed 
feedback  dynamics,  the  following  hypotheses  are  projected  for  this  research  thesis.  They 
are: 


1 .  If  information  and  knowledge  about  a  system  are  better  understood,  participant 
performance  will  improve. 

2.  If  participants  are  provided  with  a  decision  rule  that  focuses  their  attention  on 
proper  cues  and  how  to  weigh  their  importance,  their  performance  will  improve. 

3.  Participants  reporting  greater  effort  during  the  experiment  simulation  will  out¬ 
perform  those  who  do  not. 
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Research  Questions 


Given  the  stated  hypotheses,  the  following  research  questions  are  of  close  interest: 

1 .  Can  proper/adequate  knowledge  and  information  about  the  system  be  taught  to 
participants? 

2.  Can  participant  performance  be  improved  via  decision  cues  and  weights? 

3.  Can  a  participant’s  self-assessment  of  level  of  effort  be  used  to  better  determine 
their  own  experiment  performance? 
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Chapter  2 


THE  STRATEGEM-2  GAME 


Background 

In  order  to  commence,  the  reader  must  first  be  made  familiar  with  the  mechanics  of  the 
STRATEGEM-2  game.  The  term  STRATEGEM  stands  for  a  “STRATEgic  Game  for 
Educating  Managers.”  STRATEGEMs  are  a  series  of  games  produced  for  portable 
computers  and  were  developed  by  the  International  Institute  for  Applied  Systems  Analysis 
(Laxenburg,  Austria),  the  Resource  Policy  Center  (Dartmouth  College),  and  by  the  System 
Dynamics  Group  (MIT).  STRATEGEM-2  deals  with  a  micro-economy.  It  was  bom  from 
the  study  of  the  economic  long  wave,  or  Kondratiev  Cycle  (Kondratiev,  1935). 
STRATEGEM-2  first  appeared  in  the  literature  by  Sterman  and  Meadows  (1985)  and  was 
developed  to  teach  decision-making  dynamics  to  individual  players  (or  teams)  facing 
positive  feedbacks  inherent  to  a  Kondratiev  Cycle. 

Game  Play 

Briefly,  the  game  is  played  as  follows:  The  player  is  established  as  a  manager  for  a 
capital-producing  sector  of  an  economy.  Game  time  is  divided  into  two-year  intervals 
begiiming  with  year  zero  and  ending  in  year  seventy.  Thirty-five  decisions  will  be  required 
from  the  player  over  the  seventy-year  period.  The  game  board  (see  Figure  1  below),  taken 
from  the  original  Sterman  experiments  (1985,  1987,  1989a),  is  divided  into  two  sectors,  a 
capital  sector  (in  simplicity,  this  would  be  the  physical/industrial  capacity  to  produce 
consumer  goods  and  its  own  capital  goods),  and  a  goods  sector  (you  may  think  of  this  as 
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the  consumer  sector).  Orders  for  each  sector  go  into  a  “backlog  of  unfilled  orders”  area 
where  they  will  sit  awaiting  shipment  to  their  respective  sectors.  The  amount  to  be 
removed  fi’om  this  waiting  area  is  equal  to  the  capacity  of  the  capital  stock.  Additionally, 
the  capital  stock  loses  ten  percent  of  its  level  every  two  years  due  to  depreciation. 


Figure  1  -  Sterman  STRATEGEM-2  Game  Board 


In  the  Sterman  experiments  (1985,  1987,  1989a),  the  game  begins  in  equilibrium.  This 
means  that  the  capital  stock  is  at  a  level  of  500.  The  total  for  the  backlog  of  unfilled  orders 
is  500  as  well  (450  unfilled  orders  for  the  goods  sector  and  50  unfilled  orders  for  the 
capital  sector).  Finally,  a  predetermined  order  of  450  goods  sector  orders  is  displayed  for 
the  player.  This  leaves  a  single  decision  to  be  made:  How  many  orders  are  to  be  placed  in 
the  capital  sector?  The  adept  player  should  be  able  to  recognize  that  an  order  of  50  for  the 
capital  sector  will  keep  the  game  in  equilibrium.  The  reason  this  is  so  is  that  50  units  to  the 
capital  sector  will  eventually  be  used  to  replace  the  50  units  of  depreciation  the  capital 
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stock  is  scheduled  to  lose  (500  *  10%).  The  combination  of  these  50  capital  sector  orders 
with  the  established  450  goods  sector  orders  totals  500  units,  which  is  equal  to  the 
production  capacity  of  the  capital  stock.  The  capital  stock  will  then  be  able  to  produce  the 
450  orders  required  by  the  goods  sector,  and  it  will  be  able  to  produce  the  50  orders  of 
capital  to  replace  the  50  units  it  will  lose  to  depreciation.  Therefore,  the  game  will  remain 
in  equilibrium. 

On  the  upper  left  side  of  the  game  board  is  a  “thermometer-type”  display  called, 
“Fraction  of  Demand  Satisfied”  (FDS)  —  the  bar  indicates  100%  FDS  in  year  zero. 
Sterman  (1985,  1987,  1989a)  also  produces  a  “Production”  figure  in  order  to  provide 
information  to  the  player.  Production  is  calculated  as  the  minimum  of  either  capital  stock 
or  desired  production.  Plainly  stated,  industry  would  not  produce  more  than  demand 
requires.  If  the  capital  stock  were  larger  than  desired  production,  the  player  would  simply 
be  penalized  for  excess  capacity.  The  FDS  bar  is  merely  a  function  of  production  divided 
by  desired  production.  Hence,  the  only  time  FDS  is  less  than  100%  is  when  the  capital 
stock  is  less  than  the  desired  production.  Figure  2,  below,  depicts  the  STRATEGEM-2 
microeconomy  fi'om  a  “stock  and  flow”  perspective  used  by  system  dynamists  to  better 
show  the  feedback  structure  of  the  economy. 
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Capital  Sector  Orders  Goods  Sector  Goods  Sector 

.  (Exogenous) 


Player  Decision 


Figure  2  -  STRATEGEM-2  Stock  and  Flow  Structure 


As  a  final  note  on  the  game  mechanics,  after  each  round  of  play,  the  game  produces  a 
“score”  indicating  to  the  player  his  or  her  level  of  performance.  The  score  is  a  simple 
mathematical  formulation  that  keeps  an  accumulating  sum  of  the  absolute  difference 
between  the  total  desired  production  (the  total  backlog  of  unfilled  orders),  and  the 
production  capacity  of  the  capital  stock  (which  is  equal  to  the  total  of  the  capital  stock), 
and  is  divided  by  the  time  interval  (the  years  of  play).  For  example,  after  the  first  round  of 
play,  the  absolute  difference  between  desired  production  and  production  capacity  is  zero. 
Divide  that  by  2  years  and  the  score  remains  at  zero.  The  score  indicates  how  well  each 
player  can  balance  the  interactions  of  supply  and  demand.  There  is  equal  penalty  for  excess 
demand,  as  well  as  excess  supply. 
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Complexity  Added 

To  provide  complexity  to  the  game,  in  year  four,  Sterman  (1985,  1987,  1989a)  adds  a 
single  step  increase  to  orders  from  the  goods  sector.  Orders  go  up  from  450  to  500  and 
remain  at  that  level  for  the  remainder  of  the  game  (players  in  the  game  are  unav^^are  of  this 
step  increase,  or  of  its  longevity^).  The  key  is  that  the  participant  must  order  more  than  the 
depreciation  of  the  capital  stock.  The  reason;  The  capital  stock  must  be  increased  in  order 
to  meet  capacity  requirements  for  the  new  demand.  The  “gotcha”  of  the  problem  is  that  the 
player  is  more  than  likely  unaware  that  it  will  take  a  few  to  several  years  to  build  up  the 
capital  stock  to  meet  the  new  requirement.  Additionally,  the  increased  order  to  the  goods 
sector  further  complicates  the  problem  as  it  continues  to  grow  the  backlogs  of  orders  that 
need  to  be  shipped.  This  requires  that  more  capital  stock  be  ordered  so  that  the  demands  of 
the  burgeoning  backlog  of  unfilled  orders  can  be  met  -  a  classic  multiplier/investment 
accelerator  problem  (see  Figure  3  below).  This  positive  reinforcing  loop  in  the  system 
normally  forces  players  to  order  too  much  capital  stock  over  subsequent  years. 


^  This  is  a  requirement  of  the  experiment.  Otherwise,  if  the  subject  were  to  know  this  information,  he  or  she  could  possibly 
plan  accordingly  and  defeat  the  dynamics  the  system  is  trying  to  simulate. 
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Total  Demand 
for  Capital 


Orders  for  Capital 
from  the 

Consumer  Goods  Sector 


Figure  3  -  Multiplier  Accelerator  Loop 

Adapted  from  (Sterman,  1989a) 


Typically,  players  fail  to  calculate  that  with  the  new  increase  in  orders  from  the  goods 
sector  produces  a  new  equilibrium  (the  actual  new  equilibriiun  raises  from  500  to  555  - 
and  will  be  presented  as  560  in  the  game  itself  because  the  simulation  rounds  to  the  nearest 
10).  Therefore,  as  players  build  their  capital  stock  to  levels  well  above  560  (trying  to 
counteract  the  increasing  total  backlog),  they  are  slow  to  find  that  the  backlog  will  quickly 
drop  when  capacity  to  produce  is  great  and  the  orders  for  goods  sector  units  remain  at  500. 
When  participants  in  the  game  realize  that  they  now  have  too  much  capital  sector 
inventory,  they  will  tend  to  stop  ordering  all  together.  Depreciation  then  begins  to  show  its 
effect  by  lowering  the  capital  stock.  However,  the  player  soon  finds  him  or  herself  “behind 
the  power  curve”  once  again  with  not  enough  capital  stock  to  meet  the  total  demand  of  the 
economy  -  and  the  cycle  continues.  All  of  these  problems  are  the  result  of  poor 
anticipation  of  the  delays  in  the  system,  as  well  as  not  calculating  the  desired  new 
equilibrium  level. 


The  Misperception  of  Feedback  Hypothesis 


The  research  performed  by  Sterman  (1989a)  has  yielded  a  misperception  of  feedback 
hypothesis  that  attempts  to  capture  why  human  subjects  perform  poorly  in  this  simulation 
game.  Simply  stated,  the  misperception  of  feedback  hypothesis  occurs  in  two  forms.  The 

first  is  the  misperception  of  time  delays. 

“Failure  to  appreciate  time  delays  is  reflected  in  two  distinct  facets  of  the  experimental 
results.  First,  there  is  a  strong  tendency  for  subjects  to  be  overly  aggressive  in  their 
attempts  to  correct  discrepancies  between  the  desired  and  actual  capital  stock  [in  the 
game].  Second,  there  is  a  strong  tendency  to  ignore  the  time  lag  between  the  initiation  of  a 
control  action  and  its  full  effect  (Sterman,  1989a,  pg.  324).” 

The  second  form  of  the  misperception  of  feedback  hypothesis  comes  from  decisions  to 
the  environment. 

Average  behavior  on  behalf  of  participants,  “would  produce  excellent  results  if  demand 

were  exogenous  [to  the  system].  But  demand  is  not  exogenous.  The  multiplier  feedback 

causes  the  environment  to  react  endogenously  to  the  decisions  of  the  subjects.  Their 

decision  process,  however,  appears  to  be  predicated  on  an  exogenous  environment.  Thus 

many  subjects  were  surprised  that  they  did  not  receive  all  the  capital  they  ordered  as  they 

tried  to  boost  capacity.  They  were  confused  by  the  fact  that  placing  orders  to  increase 

capacity  seemed  to  worsen  the  gap  between  demand  and  supply.  And  they  were  further 

shocked  that  desired  production  suddenly  dropped  just  when  they  thought  they  had  finally 

caught  up.  These  phenomena  are  direct  consequences  of  the  multiplier  loop,  that  is, 

feedbacks  from  the  subject’s  actions  to  the  environment.  In  the  long  run,  ordering  more 

capital  does  increase  capacity,  but  in  the  short  run  it  adds  to  the  total  demand,  worsening 

the  shortfall.  Ordering  more  capital  also  raises  desired  production  further  above  capacity, 

reducing  the  fraction  of  demand  satisfied  and  delaying  delivery.  During  the  period  of 
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inadequate  capacity,  unfilled  orders  accumulate  in  the  backlog,  swelling  desired 
production.  When  capacity  finally  overtakes  desired  production,  these  accumulated  orders 
are  shipped,  and  desired  production  falls  (Sterman,  1989a,  pg.  326).” 

Therefore,  the  misperception  of  feedback  hypothesis  can  be  reduced  to  a  subject’s 
failure  to  appreciate  the  time  delay  built  into  the  game,  and  that  they  fail  to  appreciate  how 
their  decisions  are  reflected  within  the  game-playing  environment. 
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Chapter  3 


LITERATURE  REVIEW 


Overview 

Dynamic  Decision  Making,  or  DDM,  has  been  extensively  written  about  in  the 
literature  for  the  past  several  decades.  There  have  been  several  literature  reviews  written 
over  this  time  covering  the  spectrum  of  DDM  literature  (Brehmer,  1992;  Buchner,  1995; 
Funke,  1995;  Hsiao,  1999;  Kleinmuntz,  1987;  Sterman,  1994).  Of  these,  the  Hsiao  (1999) 
review  of  the  DDM  literature  is  probably  the  most  comprehensive.  He  meticulously 
reviews  and  analyzes  33  DDM  articles  from  the  period  of  1978  to  1998  (English  language 
only). 

Major  Aspects  of  the  DDM  Literature 

Hsiao  (1999)  examines  the  DDM  literature  from  the  similar  perspective  established  by 
Funke  (1995),  which  he  categorizes  relevant  variables  found  in  the  DDM  literature.  Hsiao 
(1999)  breaks  the  variables  down  into  two  major  divisions.  First,  are  the  evaluative 
variables  (read:  dependent  variables),  and  second,  are  the  predictive  variables  (read: 
independent  variables). 

Dependent  Variables 

In  the  dependent  variable  division,  Hsiao  (1999)  establishes  five  categories  that  the 
DDM  literature  has  evaluated:  Task  performance,  learning,  efforts  for  decision  making, 
quality  of  decision-making  process,  and  decision-making  architecture.  Each  of  these 
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categories  are  further  defined  by  sub-areas  indicating  the  kind  of  measures  that  the 
literature  was  being  represented  in  order  to  better  define,  or  explain,  each  category. 

Task  Performance 

In  the  first  category,  task  performance,  the  overall  idea  is  self-explanatory.  Basically, 
how  well  do  participants  perform  during  the  experiment.  Hsiao  (1999)  uses  five  sub-areas 
of  measures  for  this  category:  First,  is  the  optimizing,  maximizing,  or  minimizing, 
specified  measures  or  benchmarks,  second  is  reaching  specified  targets,  then  there  are  task 
systems  behaviors,  fourth  are  goals  combining  two  criteria,  and  finally  are  goals 
combining  greater  than  two  criteria. 

Of  these  sub-areas,  it  is  the  “optimizing,  maximizing,  or  minimizing,  specified 
measures  or  benchmarks”  that  are  important  to  the  current  research.  For  example,  studies 
that  explore  cost  (the  higher  the  cost,  the  lower  the  performance)  are  of  particular  interest 
(Diehl  &  Sterman,  1995;  Howie,  Sy,  Ford,  &  Vicente,  2000;  Richardson  and  Rohrbaugh, 
1990;  Sterman,  1987;  Sterman,  1989a;  Sterman,  1989b;  Sterman  &  Meadows,  1985). 
These  studies  specifically  consider  the  dynamic  decision  making  inherent  to  the 
STRATEGEM-2  game/model.  Subjects  within  these  studies  are  required  to  minimize 
inventory  cost  (minimizing  capital  stock  in  relation  to  demand  for  the  same  capital). 
Task-Related  Knowledge 

The  second  category,  learning,  relates  to  the  task-related  knowledge  possessed  by 
players  in  the  gaming/simulation  process.  What  is  important  is  to  determine  the  level  of 
knowledge  the  participant  has  about  the  complexity  involved  in  the  decision-making 
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process  before  and  after  the  experiment  in  order  to  determine  if  any  learning  about  the 
complexity  of  the  process  has  taken  place. 

Hsiao  (1999)  provides  five  measures  for  the  learning  category.  First,  there  is 
performance  on  preferred  tasks,  second  is  number  matching  certain  types  of  mental 
models.  Next  is  the  number  of  correctness  of  mental  models  aligned  with  heuristics  and 
goals.  Fourth,  is  to  measure  mean  scores  of  pre-game  and/or  post-game  questionnaires 
with  regard  to  procedural  task  knowledge  of  the  experiment.  Finally,  and  of  particular 
importance  to  the  current  study,  is  the  measuring  of  mean  scores  of  pre-game  and/or  post¬ 
game  questionnaires  with  regard  to  declarative  knowledge  (Howie  et  ah,  2000). 

Effort  for  Decision  Making 

The  third  category  for  dependent  variables  is  the  effort  for  decision  making.  Although 
task  performance  and  learning  are  certainly  measurable  in  a  relationship  as  an  evaluative 
variable,  an  individual’s  effort  toward  the  experimental  task  is  another  form  of  providing 
direct  observation  to  the  researcher.  This  category  may  be  subdivided  into  three  measures. 
First,  is  the  amount  of  decision  time  (how  long  does  it  take  to  make  a  decision).  Second,  is 
the  amount  of  information  use  for  specific  information  items  (is  the  participant  using  the 
information  provided  in  the  experiment).  Third,  is  the  amoxmt  of  discussion  among 
participants  (do  they  seek  each  other’s  help  when  allowed  by  the  experiment). 

The  inference  made  by  these  measures  indicates  that  there  is  greater  effort  when  there 
are  longer  decision  times,  or  greater  information  use,  or  more  collaboration  among  team 
members.  What  is  missing  fi'om  the  literature,  perhaps,  is  how  effort  can  be  measured  fi-om 
a  self-assessment  perspective.  It  is  possible,  therefore,  to  design  a  post-game  survey  that 
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can  be  used  to  determine  how  each  individual  self-assessed  their  own  level  of  effort.  Did 


the  individual  become  bored  with  the  repetitive  tasks  of  the  experiment?  Did  this  cause 
him  or  her  to  rush  to  finish?  Did  they  lose  interest?  Did  this  cause  them  to  not  pay  close 
attention?  These  are  the  questions  that  should  be  asked  in  order  to  rule  out  that  poor 
performance  was  indeed  not  influenced  by  the  “lack  of  trying.” 

Quality  of  the  Decision-making  Process 

The  fourth  category  is  related  to  the  quality  of  the  decision-making  process.  It  has  two 
measures  that  can  be  used  to  define  the  category.  First,  is  the  decision  scope.  This 
considers  the  number  of  different  decision  rules  employed  by  the  participant.  And  second, 
is  the  reliability  of  the  decisions  made.  This  involves  the  fluctuations  of  the  decisions  being 
made. 

Decision-making  Architecture 

The  final  category  for  dependent  variables  is  the  decision-making  architecture.  This 
refers  to  the  organization  of  how  decision  tasks  have  been  embedded  (Brehmer  and  Allard, 
1991).  The  measure  for  such  is  represented  by  one  thing,  the  delegation  of  decision 
making. 

In  summaiy,  the  five  categories  for  dependent  variables  deal  with  performance, 
learning,  effort,  quality,  and  architecture.  The  current  study  of  the  STRATEGEM-2  game 
will  specifically  be  concerned  with  the  performance,  learning,  and  effort  aspects  of  the 
game.  For  a  more  detailed  expose  of  this  variable,  along  with  its  associated  categories  and 
measures,  please  refer  to  Appendix  A. 
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Independent  Variables 

Hsiao  (1999)  develops  a  research  framework  of  a  dynamic  decision  making  model  (see 
Figure  4).  This  logical  framework  can  be  used  to  trace  the  decision  dynamics  from  the 
decision  makers,  through  their  game  interfaces,  into  the  realm  of  task  systems  and 
complexity  and  then  back  through  the  interface  to  the  decision  maker. 


Placing  decisions  Decisions  entering 

through  interfaces  the  task  system 


Figure  4  -  Research  Framework  of  Dynamic  Decision  Making 
Adapted  from  (Hsiao,  1 999) 

Hsiao  (1999)  posits  the  formulation  of  the  above  model  as  follows: 

“Examining  the  predictors  (independent  variables)  concerning  the  DDM 
research  would  be  a  logically  subsequent  task  after  pointing  out  those 
evaluative  criteria  (dependent  variables).  In  doing  so.  Figure  4  may  serve  as  a 
tentative  framework  containing  four  classes  of  objects  and  associated 
attributes.  First  of  all,  decision  makers,  with  the  attributes  such  as  experience, 
cognitive  style,  intelligence,  and  expertise,  should  be  an  object  of  concern  for 
they  are  the  ones  entering  a  series  of  decisions  for  a  dynamic  task.  Then 
decision-making  interfaces,  in  charge  of  entering  the  decision  to  the  task 
system,  provide  decision  makers  with  decision  outcomes  and  relevant 
information  that  may  help  them  to  make  the  next  decision.  It  is  conceivable  to 
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expect  that  the  form  and  content  of  information  display  through  the  interfaces 
should  matter  in  the  DDM  task.  A  task  system  contains  task  variables  as  well 
as  their  relationships  and  represents  a  problem  with  certain  degree  of 
complexity  in  the  real  world.  A  task  system,  usually  programmed  in  a  computer 
simulation  game,  produces  decision  outcomes.  Surrounding  the  decision 
makers,  decision-making  interfaces,  and  task  systems  are  decision-making 
enviromnents,  the  setting  in  which  decision  makers  may  receive  decision  aids 
such  as  verbal  instructions  on  task  information  and  decision  rules.  Note  that 
the  decision  aids  are  usually  perceived  through  the  decision-making  interfaces. 
According  to  Figure  4,  the  current  review  categori2es  various  predictors  of 
dynamic  decision  behavior  extracted  from  the  empirical  studies  into  three 
groups:  decision  makers’  factors,  task  complexity,  and  decision-making 
interfaces  and  environments  (Hsiao,  1999,  pg  9).” 

Hsiao  (1999)  divides  the  independent,  or  predictor,  variable  into  three  broad  categories. 
They  are:  Decision-maker  factors,  task  complexity,  and  decision-making  interfaces  and 
environments.  Each  category  is  subdivided  into  conceptual  definition  forms  with  each  of 
these  forms  being  subdivided  into  various  measures  used  by  studies  found  in  the  literature 
(Appendix  B). 

Decision-maker  Factors 

The  first  category,  decision-maker  factors,  refers  to  the  intelligence,  expertise,  skills, 

and  experience  levels  of  decision  makers.  This  category  has  four  conceptual  definitions. 

First,  is  cognitive  style,  operationalized  by  various  personality  tests  such  as  Myers-Briggs 

Type  Indicator,  Gregoric  Style  Delineator,  and  Gordon’s  Cognitive  Style  Indictor.  The 
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intent  is  to  determine  a  relationship  between  differing  personality  types  upon  the 
evaluative  variable.  The  second  conceptual  definition  concerns  task  expertise  and 
academic  training.  The  idea  suggests  that  experts  can  better  handle  complex  situations  than 
can  novices.  The  third  concept,  computing  skills,  suggest  that  computing  skill  assists 
individuals  to  overcome  the  difficulties  inherent  to  task  systems.  The  final  conceptual 
definition  for  the  decision-maker  factor  category  is  practice  and  task  experience.  The  body 
of  evidence  from  the  literature  shows  that  performance  and  learning  can  indeed  improve 
through  familiarity  and  practice  of  certain  tasks.  Of  some  interest  to  this  study  is  redundant 
testing  (with  complexity  added)  performed  by  Diehl  and  Sterman  (1995).  The  measures 
used  for  the  conceptual  definitions  described  above  can  be  found  in  Appendix  B. 

Another  decision-maker  factor  that  may  be  of  importance,  yet  not  found  in  the  DDM 
literature,  deals  with  cognitive  dissonance  theory.  The  theory,  put  forth  by  Festinger 
(1957),  simply  states  that  people  are  often  comfortable  with  their  cognitions  of  their 
surroundings,  yet,  when  confronted  with  a  cognition  that  is  in  direct  conflict  with  one’s 
own  empirical  cognition,  a  straggle  arises  within  one’s  self  that  requires  a  resolution 
(General,  2002).  The  significance  of  this  theory  is  that  when  one  is  put  into  the 
STRATEGEM-2  environment,  they  may  in  fact  be  dealing  with  competing  cognitions  that 
may  possibly  hamper  performance. 

Task  Complexity 

The  second  category  for  independent  variables  in  the  DDM  literature  is  task 
complexity,  that  is,  how  do  participants  perform?  The  idea  is  that  the  more  difficult  the 
task,  the  poorer  the  performance  will  be.  Therefore,  what  defines  task  complexity?  There 
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are  nine  conceptual  definitions  for  this  category.  They  are  explained  as  follows  (refer  to 
Appendix  B  for  the  outline  of  measures  used): 

1.  Total  number  of  variables:  The  more  variables  there  are,  the  more  difficult  the 
task. 

2.  Interaction  between  subsystems:  When  variables  have  interaction  effect  among 
themselves  (along  with  the  dependent  variable)  increases  complexity  of  the 
task. 

3.  Random  variation;  Variation  among  the  variables  increases  difficulty. 

4.  Miscellaneous  task  characteristics:  A  Hsiao  (1999)  concept  to  capture  other 
complexities  not  otherwise  outlined  here. 

5.  Time  delay  and  lagged  effects:  This  concept  is  most  important  to  the 
STRATEGEM-2  simulation  and  is  best  captured  by  Hsiao  (1999).  “The  DDM 
research  has  been  focusing  on  additional  complexity  predictors  mostly  unique 
in  dynamic  task  environments.  Lagged  effects  of  decisions  refer  to  a  common 
phenomenon  that  a  previous  decision  at  time  t-1  does  not  always  take  effect 
immediately  at  time  t.  Subjects  may  not  see  the  effect  until  time  t+1  or  later 
periods  (e.g.,  Berry  and  Broadbent,  1988);  or  they  may  not  even  perceive  the 
effect  at  all  due  to  other  task  complexity  factors.  Lagged  effects  result  from 
formulation,  of  which  time  delay  is  the  most  noticeable  one  (e.g.,  Sterman, 
1989a,  1989b,  Diehl  etal.,  1995)  (pg.  10).” 
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6.  Effectiveness  of  decisions  on  outcomes:  These  refer  to  the  participant’s  ability 
to  interpret  lagged  effects. 

7.  Frequency  of  oscillation:  This  refers  to  the  stability  of  the  system.  The  more 
unstable,  the  more  difficult  the  task. 

8.  Positive  feedback  and  gains:  System  dynamicists  have  been  able  to  account  for 
many  of  the  instabilities  observed  in  the  previous  concept  through  algebraic 
formulations  of  positive  feedback  loops  within  a  system  (e.g.  Sterman,  1 989a, 
Diehl  and  Sterman,  1995).  In  other  words,  decisions  upon  one  or  more 
variables  will  effect  other  parts  of  a  system  that  will,  in  turn,  affect  future 
decisions.  Another  example  is  how  “word  of  mouth”  can  effect  positive  gains 
in  a  market  strategy  game  (Paich  and  Sterman,  1993). 

9.  Real-time  simulation  tasks:  The  body  of  work  in  this  area  deals  with  how 
complexity  is  added  when  the  experiment  is  performed  in  a  real-time,  clock- 
driven,  environment.  Brehmer  (1992),  with  his  fire  fighting  experiments,  are 
the  cornerstone  for  real-time  complexities  in  the  DDM  literature. 

Decision-making  Interfaces 

The  final  category  for  independent  variables  is  decision-making  interfaces  and 
environments.  This  captures  the  factors  that  are  in  between  decision  makers  and  task 
systems.  Hsiao  (1999)  further  defined  this  category  into  eleven  conceptual  definitions. 
Refer  again  to  Appendix  B  for  specific  measures  and  studies  applied  against  the  following 
concepts: 
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1.  Heuristics  (decision  rules)  built  into  task  systems:  Rules  that  provide  explicit 
instructions  on  what  a  decision  should  be  based  on  previous  outcomes.  The 
Richardson  and  Rohrbaugh  (1990)  decision  mle  falls  under  this  concept. 

2.  Modes  of  learning  induced  by  lagged  effects:  “Specifically,  when  subjects 
perform  a  task  without  any  lagged  effects,  they  tend  to  concentrate  on 
developing  the  relationships  of  the  variables  they  think  important. 
Comparatively,  when  subjects  experience  a  task  with  lagged  effects,  they  tend 
to  be  impressed  with  the  cases  of  individually  paired  decision-outcome 
matches,  without  systematically  forming  variables’  relationship.  The  former, 
termed  the  selective  mode  of  learning  (or  explicit  learning  by  other  authors, 
e.g..  Berry  and  Broadbent,  1988),  enables  subjects  to  acquire  verbalized 
knowledge  which  can  easily  be  measured  by  post-task  questions.  The  latter, 
termed  the  unselective  (or  implicit)  mode,  may  not  be  verbalized  but  still 
function  in  certain  situations  (Hsiao,  1999,  pg.  12).”  Additionally,  this  refers  to 
suggesting  that  participants  pay  particular  attention  to  key  variables, 
instructions  on  variables’  relationships,  important  feedback  loops,  and  decision 
effectiveness  that  can  be  helpful  (Berry  and  Broadbent,  1988;  Wang,  1994). 

3.  Heuristics-induced  goal  setting  that  subjects  receive  through  verbal 
instructions:  These  heuristics  attempt  to  influence  decisions,  and  therefore, 
performance  and  learning.  The  Sterman  (1989a)  decision  rule  is  applied  here. 

4.  Task  property,  strategies,  and  heuristics  that  subjects  receive  through  verbal 

instructions:  The  importance  here  is  to  provide  aids  to  participants  to  allow 
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them  to  understand  task  structure  and  the  relationship  among  variables.  The 
STRATEGEM-2  game  studies  performed  by  Richardson  and  Rohrbaugh 
(1990)  fall  into  this  precept. 

5.  Concurrent  verbalization  and  thinking  aloud:  A  concept  that  requires  subjects 
to  verbalize  aloud  their  decision  strategies  and  rules  while  making  decisions. 
The  idea  is  that  through  verbalization,  participants  will  improve  their  task- 
related  knowledge. 

6.  Increasing  task  salience:  In  order  to  increase  task  importance,  Berry  and 
Broadbent  (1988)  and  Wang  (1994)  instmct  participants  about  task  structure 
information  and  how  lagged  effects  may  produce  various  outcomes.  They 
found  that  their  decision-aid  supports  task  performance  abilities. 

7.  Degree  of  decision  precision  required:  In  this  concept,  the  participant  is 
conditioned  to  learn  better  -  to  produce  results  at  the  “first  decimal  place  versus 
the  whole  number.” 

8.  Learning  inducement:  Produces  subjects  to  search  for  better  understanding  of 
task  structure  over  task  performance. 

9.  Contents  of  information  display:  The  exploration  of  this  concept  tests  “what 
information  is  helpful?” 

10.  Forms  of  information  display:  Critical  to  the  current  study,  this  concept 
considers  very  closely  the  work  established  by  Sengupta  and  Abdel-Hamid 
(1993),  where  they  “base  their  research  design  on  the  theory  of  information 
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feedback  and  provide  subjects  with  three  types  of  computer  information 
feedback:  outcome  feedback,  cognitive  feedback,  and  feedforward.  Outcome 
feedback  indicates  online  numerical  reports  for  important  state  variables  of  the 
software  project  task.  Subjects  receiving  cognitive  feedback  have  access  to 
online  time  plots  containing  the  patterns  of  relevant  variables  and  a  tabular 
summary  of  these  cues.  Whereas  outcome  and  cognitive  feedback  are  always 
available  on  computer  screens,  feedforward  is  conveyed  by  an  hour-long 
training  session  prior  to  the  task,  same  as  those  decision  heuristics  described 
above  (Hsiao,  1999,  pg  14).” 

“Decision  rules  and  relevant  cues  have  been  incorporated  in  Richardson 
and  Rohrbaugh’s  study  (1990)  where  a  group  of  subjects  are  provided  with 
numerical  weights  of  the  three  cues  and  the  other  group  with  the  same  plus  a 
simple  decision  mle  to  transform  the  cues  into  a  decision.  Compared  with  the 
preceding  instructions  of  decision  rules  and  task  property,  the  information 
display  issue  appears  to  be  left  imexplored  (Hsiao,  1999,  pg  14-15).  This  does 
not  consider  the  initial  foray  into  this  area  by  Howie  and  others  (2000). 

1 1.  Decision-making  architectures:  This  concept  purports  that  the  command 
stmcture  of  a  decision  affects  task  performance.  This  can  be  formed  as  a 
networked  architecture  (open  communication  among  participants),  or  a 
command-down  architecture  where  participants  receive  orders  from  a  single 
player. 
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For  a  more  detailed  expose  of  the  predictor  variable,  along  with  its  associated 
categories  and  measures,  please  refer  to  Appendix  B. 

Original  STRATEGEM-2  Studies 

Sterman  (1989a)  examines  human  subject  dynamic  decision  making  using  the 
STRATEGEM-2  computer  game,  and  he  bases  this  research  on  previous  studies  of  his 
own  from  1987  and  1985.  Subjects  are  required  to  make  a  single  decision  on  capital  orders 
within  a  dynamic  system.  Tasks  involve  feedback  delays,  nonlinearity  of  system  processes, 
positive  feedback  loops,  and  multiple  cues.  Sterman  (1989a)  viewed  the  problem  as  being 
framed  by  there  being  so  little  corporate  epistemological*  work  relating  decision-maker 
behaviors  to  large  organization  dynamics. 

From  the  results  of  Sterman’s  (1987,  1989a)  work,  he  develops  a  “decision  rule”  that 
purports  to  capture  the  decision-making  behavior  of  people  playing  the  game.  From  this, 
he  simply  recommends  a  three-cue  task  to  his  subjects  in  that  they  “order  enough  to 
replace  depreciation,  adjust  it  by  some  fraction  of  the  discrepancy  between  the  desired  and 
actual  levels  of  capacity,  and  don't  forget  to  take  the  supply  line  of  previous  orders  into 
accoimt"  (Sterman,  1987,  p.  1588).  He  breaks  this  rule  out  into  five  different  equations  that 
logically  capture  how  participants  normally  play  the  game. 

Sterman  (1989a)  also  displays  an  “optimal”  solution  to  the  game  (Appendix  C).  A 
solution  that  has  the  game  back  into  perfect  equilibrium  within  five  moves,  or  decisions 
(ten  years  of  game  time),  following  the  step  increase  in  goods  orders  in  year  four.  A  final 


*  From  epistemoldgf.  A  branch  of  philosophy  that  investigates  the  origin,  nature,  methods,  and  limits  of  knowledge. 
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score  of  19  is  produced  using  the  “optimal”  solution.  Unfortunately,  Sterman  (1989a)  does 
not  provide  the  mechanics  on  how  to  determine  the  optimal  solution. 

Sterman  (1989a)  hypothesizes  that  his  subjects  would  make  decisions  based  on 
anchoring  and  adjustment  heuristics  as  suggested  by  his  decision  rule  and  that  they  would 
be  motivated  by  the  observation  that  the  complexity  of  determining  the  optimal  rule 
(whatever  that  is)  would  be  overwhelming  to  their  abilities.  In  other  words,  Sterman 
(1989a)  was  predicting  that  players  of  the  game  would  not  perform  well. 

The  results  of  his  findings,  from  a  field  of  49  participants,  produced  a  mean  score  of 
591.  However,  the  top  half  of  the  best  participants  had  a  score  less  than  300  (the  top  player 
produced  a  score  of  77).  Yet,  as  Sterman  (1989a)  observes,  none  were  even  close  to  the 
optimal  score  of  19.  He  was  forthright  when  analyzing  the  (apparent)  success  of  his 
experiment.  He  stated  that:  “The  experimental  results  suggest  that  subjects  do  not  behave 
optimally  even  when  provided  with  perfect  information  and  knowledge  of  the  system...” 
Additionally,  “the  results  reveal  several  misperceptions  of  feedback:  many  subjects  fail  to 
account  adequately  for  the  delay  between  their  own  decisions  and  the  environment” 
(Sterman,  1989a,  p.  329). 

Early  Critique  of  STRATEGEM-2  Findings 

Richardson  and  Rohrbaugh  (1990)  posited  the  following:  “How  would  players  perform 

if  the  computer  screen  directly  provided  them  with  the  cues  appropriate  for  the  task?  What 

effect  would  different  forms  of  cue  presentation  have  on  cognitive  learning?  These 

questions  are  important  because  they  may  reveal  an  alternative  explanation  for  the 

misperceptions  and  dysfunctional  behaviors  found  by  Sterman  (1989a).  We  hypothesize 
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that  the  form  of  cue  presentation  used  for  the  study  of  decision  making  in  dynamic 
environments  will  have  a  significant  effect  on  results  (pg.  464).”  In  order  to  analyze  their 
hypothesis,  they  developed  a  three-condition  experiment.  Although  the  first  two  conditions 
of  the  experiment  are  important  in  their  research  design,  they  will  not  be  discussed  here  in 
order  that  full  attention  can  be  given  to  the  third  condition,  which  represents  the  crux  of 
their  hypothesis. 

The  STRATEGEM-2  computer  game  was  modified  by  Richardson  and  Rohrbaugh 
(1990)  to  accommodate  and  correct  what  they  viewed  as  pitfalls  of  the  Sterman  (1987, 
1989a)  experiments.  They  felt  that  two  cues,  required  for  optimal  play  of  the  game, 
specifically,  depreciation  and  shortfall,  were  not  explicit  on  the  (Sterman,  1987,  1989a) 
computer  screen  -  they  were  assumed  to  be  calculable  by  the  player.  Additionally, 
Richardson  and  Rohrbaugh  (1990)  suggested  that  players  had  to  be  sophisticated  in  order 
to  use  the  Fraction  of  Demand  Satisfied  bar  graph  on  the  Sterman  (1987,  1989a)  screen. 
This  is  because  players  would  have  to  realize  that  the  “delay”  or  “production  capacity”  of 
their  orders  is  the  inverse  of  the  FDS.  They  concluded  that  a  better  ordering  strategy 
designed  for  the  computer  screen  can  be  established. 

The  Richardson  and  Rohrbaugh  Decision  Rule 

The  game  board  used  for  the  Richardson  and  Rohrbaugh  (1990)  experiment  is 
replicated  in  Figure  5  (below).  One  can  see  that  it  has  several  changes  fi-om  the  one  used  in 
the  Sterman  (1989a)  experiment  (Figure  1).  First,  the  depreciation  of  capital  is  explicitly 
shown  (50  units).  Second,  production  has  been  replaced  with  shortfall  (the  desired 
production  minus  the  capital  stock).  Finally,  a  decision  mle  was  put  in  place  of  the 
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Steiman  FDS  bar  graph.  The  Richardson  and  Rohrbaugh  decision  rule  specifies  that  the 
player  would  want  to  perform  the  following  in  year  zero:  1)  Take  the  current  depreciation 
of  50  units  and  multiply  it  times  2  (for  100).  Add  to  that,  2)  Take  the  shortfall,  currently  0 
and  divide  by  2  (for  0),  and  finally,  3)  Subtract  the  current  capital  backlog  of  50.  The  rule 
proposed  here  by  Richardson  and  Rohrbaugh  (1990)  would  then  prompt  the  player  to  order 
50  units  in  year  zero,  which,  of  course,  will  keep  the  game  in  equilibrium. 


—  To  Order  ~ 

Plan  in  advance  to 
replace  depreciation  loss 
DEPRECIATION: 

+  50  X  2 

Reconcile  desired  production 
with  actual  capital  stock 

SHORTFALL:  +0  +  2 

Adjust  for  prior  orders 
not  yet  filled 

BACKLOG: 

-  50 

Year  0 


Depreciation 
(10%  of  Capital  Stock) 


Shortfall 

(Desired  Production  •  Capital  Stock) 
0 


New  Orders 
Capital  Sector 


Desired  Production 
500 

Backlog  of  Unfilled  Orders 


Capita) 

Sector 

50 


Goods 

Sector 

450 


Figure  5  -  The  Richardson  and  Rohrbaugh  SRATEGEM-2  Game  Interface 


Given  the  Richardson  and  Rohrbaugh  decision  mle,  did  their  participants  have  to 
strictly  adhere  to  this  rule?  No.  As  in  the  Sterman  (1987,  1989a)  experiment,  players  were 
always  allowed  to  decide  as  best  as  they  saw  fit.  The  proposed  rule  provided  by 
Richardson  and  Rohrbaugh  (1990)  was  simply  there  to  assist  the  decision  makers  in 
dealing  with  the  complex  task  of  balancing  supply  and  demand  in  the  micro-economy. 
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The  Richardson  and  Rohrbaugh  (1990)  experiment  had  only  18  participants  of  which 
only  6  were  randomly  assigned  to  the  condition  described  above.  The  authors  received 
mixed  results  in  their  study.  However,  the  results  do  not  go  without  adequate  explanation. 
For  example,  they  were  able  to  determine  that  one  half  of  the  participants  were  able  to 
significantly  perform  to  the  level  they  had  hypothesized.  From  the  standpoint  as  an  outside 
observer,  it  would  additionally  be  concluded  that  the  small  sample  size  contributed  to  their 
sub-optimal  findings.  Richardson  and  Rohrbaugh  (1990)  did  determine  that  when  the 
subjects  improved  their  consistency  in  information  usage  there  is  a  Imk  to  maintaining 
stability  in  the  system.  Richardson  and  Rohrbaugh  (1990),  therefore,  established  that  there 
is  great  room  for  additional  research  in  this  area. 

Secondary  Critique  of  STRATEGEM-2  Findings 

In  the  Howie  and  others  (2000)  paper,  the  work  of  Richardson  and  Rohrbaugh  (1990) 
was  partially  reexamined.  The  authors  took  to  task  the  primary  assumption  of  Sterman 
(1989a)  that  the  players  of  the  STRATEGEM-2  game  have  “perfect  information”  of  the 
system.  They  viewed  the  misperception  of  feedback  hypothesis  included  a  certain  degree 
of  pessimism  of  the  human  endeavor  within  dynamic  systems.  From  the  most  pessimistic 
angle,  they  quote  Sterman  as  follows:  “...improved  performance  can  only  be  achieved 
through  automated  decision  support  because  human  dynamic  decision  making  is  bound  to 
be  poor  because  of  ‘a  fundamental  bound  on  human  rationality’(Howie  et  al.,  2000,  pg. 
152).”  A  less  pessimistic  examination  of  the  misperception  of  feedback  hypothesis  is  also 
presented  in  “that  people  do  not  do  well  because  their  knowledge  of  system  stmctures  is 
less  than  perfect.  In  this  case,  poor  performance  is  caused  by  lack  of  knowledge  rather  than 
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some  fundamental  psychological  limitation  (Howie  et  al.,  2000,  pg.  152).”  And  finally,  the 
authors  conclude  with  a  more  optimistic  approach  by  stating  “that  people  perform  poorly 
because  the  information  they  need  to  do  the  task  is  not  presented  in  the  computer  display 
that  they  have  available  to  them.  In  this  case,  the  poor  performance  is  caused  by  a  lack  of 
information  rather  than  a  lack  of  knowledge  or  a  fundamental  psychological  limitation  (pg. 
152-153).” 

The  thrust  of  Howie  and  others’  (2000)  article  is  that  players  in  the  STRATEGEM-2 
game  do  not  have  perfect  knowledge  or  perfect  information,  as  has  been  put  forth  in 
misperception  of  feedback  hypothesis  published  studies.  For  example,  according  to 
Sterman  (1987),  participants  had  “perfect  knowledge  of  the  system  stmcture  and  perfect 
information  (pg.  1587).”  To  the  contrary,  Howie  and  others  (2000),  was  the  first  to  test 
player  knowledge  of  the  game  itself  They  provided  a  pre-game  and  post-game  test 
covering  the  declarative  knowledge  of  the  simulation.  Not  one  subject  received  a  perfect 
score  in  either  phase  of  testing.  In  fact,  the  test  scores  were  rather  low  from  the  pre-game 
phase.  Scores  in  the  post-game  phase  did,  however,  show  improvement  over  pre-game 
scores,  which  indicate  that  some  knowledge  had  been  gained  from  exposure  to  playing  the 
game.  The  important  item  that  Howie  and  others  (2000)  was  showing  was  that  participants 
did  not  have  “perfect  knowledge”  of  the  game. 

Similar  to  Richardson  and  Rohrbaugh  (1990),  Howie  and  others  (2000)  focused  on  the 
interface  used  during  STRATEGEM-2  play.  They  developed  a  completely  new 
Windows®-based  interface.  An  interface  that,  in  their  argument,  reflects  proper  design 
according  to  human-computer  interaction  principles  that  is  in  that  body  of  literature.  They 
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hypothesized  that  subjects  playing  the  STRATEGEM-2  game  with  their  newly  designed 
interface,  would  perform  better  than  participants  playing  the  same  game  with  the  older 
interface  developed  by  Sterman  (1989a).  See  Appendbc  F  for  various  displays  of  the 
Howie  interface. 

The  results  obtained  by  Howie  and  others  (2000),  like  Richardson  and  Rohrbaugh 
(1990),  were  mixed.  In  the  two  trials  that  were  performed,  the  group  using  the  old  interface 
performed  better  than  the  group  using  the  newly  designed  interface,  however,  the  standard 
error  of  their  mean  scores  did  overlap  each  other,  which  meant  that  some  people  in  each 
group  were  actually  performing  on  the  same  level  as  each  other.  In  the  second  trial,  the 
new  interface  group  far  outperformed  the  old  interface  group.  This  time,  there  was  no 
overlapping  of  scores  -  the  results  were  considered  to  be  influenced  by  the  new  interface. 

Other  Pertinent  Studies 

There  are  two  additional  studies  that  have  particular  weight  to  the  current  research. 
First,  is  from  Kim  Vicente  (1996),  where  he  demonstrates  that  computer  interface  design 
can  have  positive,  or  negative,  impacts  on  human  performance.  Second,  the  research  by 
Sengupta  and  Abdel-Hamid  (1993)  purports  that  subjects  provided  with  cognitive 
feedback,  and  subjects  provided  with  feedforward  show  better  performance  than  with 
subjects  who  are  only  provided  outcome  feedback  information. 

Vicente  (1996)  explored  the  possibilities  of  enhancing  ecological  interface  design 
(EID)  as  a  framework  for  complex  human-machine  systems.  He  proposed  that  the  EID 
framework  consisted  of  three  principles  (each  intended  to  support  a  given  level  of 
cognitive  control),  they  are: 
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1.  “Knowledge-based  behavior  -  this  represents  the  externalized  mental  model  that 
will  support  analytical  problem  solving 

2.  “Rule-based  behavior  —  this  provides  a  consistent  one-to-one  mapping  between  the 
work  domain  constraints  and  the  cues  provided  by  the  interface 

3.  “Skill-based  behavior  -  this  supports  interaction  via  time-space  signals;  the 
operator  should  be  able  to  act  directly  on  the  display  (pg.  253).” 

In  his  study,  Vicente  (1996)  simulated  the  dynamics  of  thermal-hydraulic  process- 
control  rooms  (regulating  the  temperature  and  volumes  of  hydro-electric  power  facilities). 
The  aim  of  decision  makers  is  to  keep  water  temperatures  of  two  water  reservoirs  at  a 
constant  temperature,  while  at  the  same  time,  keeping  each  reservoir  at  a  given  water  level. 

Vicente  (1996)  acknowledged  that  in  system  dynamics  studies,  researchers  have 

frequently  concluded  that  participants  are  severely  impaired  in  their  ability  to  cope 

effectively  with  complex  systems  (for  example:  Sterman,  1989a).  He  posits  that  one 

potential  explanation  for  this  behavior  is  that  research  subjects  did  not  have  enough 

practice  with  a  given  simulation  in  order  to  adapt  to  its  dynamics.  Additionally,  he  further 

states  another  explanation  in  that,  “the  findings  paint  a  very  unflattering  picture  of  people’s 

capabilities  to  engage  in  dynamic  decision  making  in  complex  systems  which  may,  in  part, 

be  due  to  the  impoverished  interfaces  used  in  those  experiments.  In  fact,  some  authors 

explicitly  refer  to  opaqueness  (Brehmer,  1992)  or  lack  of  transparency  (Domer,  1987),  as 

characteristics  of  dynamic  decision-making  problems.  However,  the  research  reviewed  [in 

this  study]  shows  that  opaqueness  is  a  property  of  an  interface,  not  an  inherent  property  of 

complex  systems  (Vicente,  1996,  pg.  275).”  And,  “by  accepting  existing  interfaces  as  an 
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unalterable  given,  one  might  be  led  to  the  unqualified  conclusion  that  people  are  poor 
decision  makers  in  complex  systems  (pg.  277).” 

The  work  of  Vicente  (1996)  was  critical  in  developing  a  new  STRATEGEM-2 
interface  as  put  forth  by  Howie  and  others  (2000).  This  is  also  the  reason  the  current 
research  must  incorporate  the  new  interface  design. 

Kishore  Sengupta  and  Tarek  Abdel-Hamid  (1993)  develop  alternative  conceptions  of 
feedback  in  dynamic  decision  environments:  They  experimentally  investigate  the 
relationships  of  outcome  feedback,  cognitive  feedback,  and  feedforward  in  a  system 
dynamics  simulation  game  where  human  subjects  perform  duties  as  program  managers  by 
making  decisions  over  the  life  of  computer  software  projects.  The  concepts  of  each  are  as 
follows: 

1.  Outcome  Feedback:  “It  has  been  argued  that  in  dynamic  environments,  outcome 
feedback  acquires  the  property  of  being  corrective  feedback  in  that  it  permits 
adjustments  to  the  general  direction  of  judgment  (Hogarth,  1981).  Decision  makers 
can,  therefore,  rely  on  outcome  feedback  through  a  judgment-action-feedback  loop 
to  make  effective  decisions.  Thus,  a  decision  maker  acting  at  time  t  has  the  benefit 
of  outcome  feedback  from  time  t-\,  enabling  the  individual  to  make  appropriate 
changes  in  decision  strategy  (Sengupta  and  Abdel-Hamid,  1993,  pg.  41 1).” 

2.  Cognitive  Feedback:  “Is  conceptualized  as  information  provided  to  a  decision 
maker  about  (a)  the  relations  in  the  decision  environment,  (b)  relations  perceived 
by  the  person  about  the  environment,  and  (c)  relations  between  the  environment 

and  the  person’s  perception  (Sengupta  and  Abdel-Hamid,  1993,  pg.  412).” 
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Probably  the  most  pertinent  method  for  these  conceptualizations  to  occur  would  be 
through  task  information  and  cognitive  information.  “Task  information  enables  a 
decision  maker  to  learn  more  about  the  environment,  e.g.  the  relationship  among 
the  cues  comprising  a  task.  Cognitive  information  enables  an  individual  to  gain 
greater  insight  into  his/her  decision  strategy,  e.g.  through  information  on  weights 
accorded  by  the  person  to  various  cues  (pg.  413).” 

3.  Feedforward:  “Attempts  to  improve  an  individual’s  decision  quality  by  providing 
him/her  with  a  model  of  the  task  prior  to  performing  the  task*.  .  .  Feedforward 
reduces  the  cognitive  load  on  a  subject  because  a  large  amount  of  the  information 
the  subject  would  have  to  infer  from  feedback  is  already  transmitted  through  prior 
instmctions. . .  [The  benefit  of  this  method  is  that  it]  enables  the  decision  maker  to 
understand  the  key  relationships  and  time  lags  that  cannot  be  inferred  from 
outcome  feedback  alone.  Feedforward  can  thus  serve  as  an  effective  method  of 
planning  an  overall  strategy  (Sengupta  and  Abdel-Hamid,  1993,  pg.  413-414).” 

One  of  the  largest  problems  facing  people  in  dynamic  environments  deals  with 
outcome  feedback.  Simply  stated,  outcome  feedback  does  not  provide  enough  information 
in  order  for  decision  makers  to  form  adequate  models  of  system  behavior.  The  concepts  of 
cognitive  feedback  and  feedforward  provide  added  insight  and  assistance  to  the  decision 
maker.  It  is  this  researcher’s  position  that  cognitive  feedback  along  with  feedforward 


*  “The  term  feedforward  is  issued  here  in  the  specific  sense  of  conveying  task  information  to  a  decision  maker,  and  should 
not  be  confused  with  the  manner  in  which  it  is  used  by  researchers  in  human-computer  interaction  (Sengupta  and 
Abdel-Hamid,  1993,  pg.  413).” 
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provides  the  best  environment  in  which  to  make  decisions  when  faced  with  complex 
systems.  The  Richardson  and  Rohrbaugh  (1990)  study  inherently  proposed  such  a  design. 

Designing  a  Tutorial  Instruction  Set 

As  suggested  in  Chapter  1,  oftentimes  researchers  do  not  pay  close  enough  attention  to 
the  experimental  “setup”  or  instruction  of  human  subjects.  As  will  be  seen  in  Chapter  4, 
Methodology,  a  computer-based  on-screen  tutorial  is  planned  for  the  instmctional  phase  of 
the  experiment.  In  order  to  develop  this  portion  of  the  experiment,  a  review  of  the  literature 
is  necessary  in  order  to  find  common  elements  of  good  tutorial  design. 

Computer-based  instruction,  also  known  as:  computer-aided/assisted  instmction, 
multimedia  interface,  interactive  multimedia  instruction,  and  intelligent  tutoring  systems 
(just  to  name  a  few),  is  a  relatively  new  phenomenon  in  educating  persons  and  has 
experienced  its  largest  developments  in  only  the  past  two  decades.  Kemp  and  Dayton 
(1985)  were  early  architects  of  computer-based  instruction  (CBI),  and  they  recognized  two 
distinct  methods  of  teaching  using  a  computer  for  tutorial  purposes.  The  first  of  these 
methods  presents  information  to  the  user  in  a  fixed  sequence.  This  “linear  program”  does 
not  allow  for  differences  among  individual  users.  There  may  be  some  advantages  to  this 
type  of  tutorial  programming.  For  example,  if  one  wanted  to  research  the  individual 
differences  in  learning,  this  method  would  be  more  appropriate.  The  second  method 
provides  options  to  be  chosen  on  behalf  of  the  learner,  allowing  him,  or  her,  to  follow 
various  paths  of  instruction.  This  is  called  a  “branching  program”  and  can  result  in  better 
individualized  learning. 
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Kemp  and  Dayton  (1985)  identified  four  major  areas  that  CBI  can  be  used  to  enhance 
individualized  learning.  They  are: 

1 .  Drill  and  practice:  Provides  practice  for  reinforcement  of  a  concept  or  skill.  The 
computer  is  used  to  provide  a  series  of  questions  or  exercises  (similar  to  those 
found  in  a  workbook). 

2.  Tutorials:  They  attempt  to  take  the  place  of  a  human  tutor.  For  example,  they 
can  be  used  to  pose  problems  requiring  a  correct  response  from  the  participant, 
which  navigates  the  learner  to  another  block  of  instruction  or  to  a  remedial 
training  block  (depending  on  a  correct  or  incorrect  response). 

3.  Simulations:  These  are  used  to  imitate  dynamic  processes  or  systems.  For 
example,  they  can  be  used  to  navigate  a  sea-going  vessel,  manipulating  an 
economy,  or  to  operate  the  controls  of  some  sort  of  manufacturing  machine.  In 
essence,  simulations  are  used  in  order  to  provide  the  user  an  experience  of  real 
world  without  having  to  endure  a  real  world  consequence  of  a  ship  sinking,  or 
an  economic  depression,  or  industrial  accident.  Additionally,  these  simulations 
significantly  reduce  training  costs  while  at  the  same  time,  reduces  the  timeline 
required  to  gain  the  experience. 

4.  Games:  If  designed  properly,  games  can  be  used  to  teach  while  taking 
advantage  of  the  participant’s  competitive  nature.  The  motivation  to  “win”  in 
turn  increases  the  learning  outcome. 
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With  regard  to  the  current  experiment,  the  intent  is  to  develop  the  tutorial  portion. 
Additionally,  STRATEGEM-2  is  not  only  a  simulation,  but  it  serves  as  a  game  as  well. 
However,  care  must  be  taken  when  developing  an  instruction  set  for  such  an  experiment. 
“In  multimedia  instruction,  features  of  games  and  simulations  are  often  combined,  as  both 
approaches  offer  highly  motivational  and  potentially  relevant  environments.  However,  one 
caution  must  be  underscored.  Many  simulations  and  games  may  not  emphasize 
prescriptive  instmction;  the  primary  purpose  of  many  games  and  simulations  is 
entertainment  or  vicarious  experience,  with  learning  as  a  convenient  by-product. 
Prescriptive  instruction  requires  learning  to  be  at  the  heart  of  the  product,  with  the  goals 
and  parameters  clearly  defined  (Schwier  and  Misanchuk,  1993).” 

In  developing  a  tutorial  strategy,  Halff  (1988)  espouses  that  the  methods  used  to 
present  material  to  the  participant,  depends  on  the  instructional  objectives  and  subject 
matter.  He  uses  a  dialogue  strategy  put  forth  by  Collins  and  Stevens  (1982)  and  is 
displayed  in  Table  1. 


Instructional  Objective 

Strategies 

Teach  facts  and  concepts 

Elicit  fact  or  concept 

Explain  fact  or  concept 

Teach  rules  and  relations 

Case  selection  strategies 

Entrapment 

Teach  induction  skills 

Exercises  and  examples 
oriented  to  subskills 

Table  1  -  Tutorial  Dialogue  Strategies  for  Different  Instructional  Objectives 


“Teaching  of  facts  and  concepts  is  accomplished  by  asking  for  or 
explaining  the  material.  The  decision  to  ask  or  tell  is  made  on  the  basis  of 
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the  importance  of  the  material  and  the  student's  knowledge  thereof. 
Teaching  of  rules  in  tutorial  sessions  usually  involves  inducing  the  student 
to  consider  the  relevant  data  and  to  formulate  the  rule.  This  can  be  done  by 
presenting  case  data  that  makes  the  rule  clear  or  by  entrapment  strategies 
that  enable  the  student  to  eliminate  incorrect  versions  of  the  rule.  Skills  for 
deriving  [inducing]  rules  are  taught  as  procedures.  These  procedures  are 
broken  down  into  their  components  (e.g.,  listing  factors,  generating  cases 
to  specification),  and  exercises  and  examples  are  provided  that  address 
each  subskill  (Halff,  1988,  pg.  90).” 

Soulier  (1988)  takes  computer-based  instruction  to  a  higher  level.  He  introduces  the 
concept  of  management  flumes  designed  to  aide  the  participant  in  his,  or  her,  learning. 
Generically,  CBI  fi'ames  are  considered  important  in  the  instructional  process,  however, 
they  do  not  teach,  per  se.  Rather,  Soulier  (1988)  proposes  “dialog  fi-ames”  and  “criterion 
fi-ames”  to  better  help  the  learner.  “Dialog  fi-ames  present  information  to  the  learner,  as 
well  as  carry  out  an  interactive  dialogue/feedback  between  the  learner  and  the  computer. 
Criterion  fiames  assess  learning  performance  and  provide  feedback  on  results  and  follow¬ 
up  activities  (pg.  141).”  Although  not  considered  totally  relevant  to  the  current  research 
problem,  Soulier’s  insights  may  provide  some  direction  towards  development  of  an  on¬ 
screen  tutorial  for  participants  of  the  STRATEGEM-2  game. 

In  developing  computer-based  instruction,  attention  must  be  paid  to  various  display 
properties  necessary  to  convey  correct,  and  intended,  information.  One  of  the  most  critical 
to  these  properties  is  the  passage  length  (Steinberg,  1990). 
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""Passage  length.  The  length  of  a  passage  is  of  special  concern  in  CAI 
[computer-aided  instruction].  When  talking,  the  number  of  words  used  in 
an  explanation  is  not  restricted  by  space.  In  the  classroom,  an  instructor 
may  discuss  a  subject  at  some  length,  constrained  only  by  time  and  his 
ability  to  maintain  students'  attention.  Textbook  writers  may  also  present 
extensive  discussions,  limited  only  by  publishers'  page  restrictions.  In  CAI, 
it  is  not  feasible  to  present  a  great  deal  of  verbal  discourse.  For  some 
unexplained  reason  people  read  more  slowly  when  text  is  presented  on  a 
display  screen.  Furthermore,  students  do  not  tolerate  a  computer  program 
that  is  essentially  an  electronic  page-turner.  Perhaps  this  is  because  they  are 
still  unaccustomed  to  using  a  computer  program  for  extended  reading.  It 
may  be  due  to  the  expectation  that  a  computer  program  should  be  highly 
interactive  (Steinberg,  1990,  pg.  84-85).” 

The  importance  of  Steinberg’s  (1990)  “passage”  prescription  reflects  upon  the  notion 
that  the  presentation  must  “get  to  the  point.”  To  belabor  the  participant  with  lengthy 
dialogs  may  detract,  rather  than,  enhance  the  learning  process.  The  important  point  for 
tutorial  designer’s  to  remember  is  try  and  capture  specific  learning  points  onto  a  single 
screen,  or  frame. 

Although  several  authors  have  defined  some  of  the  necessary  tools  in  developing 
various  forms  of  CBI,  Price  (1991)  brings  the  CBI  designer  in  line  with  “goals  and 
objectives”  processes  necessary  to  properly  develop  computer  tutorials.  For  example.  Price 
critically  delineates  that  goals  cannot  be  generalized;  rather,  they  must  be  precisely  stated 
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as  clearly  as  possible  (avoiding  ambiguity  that  would  leave  questions  in  the  mind  of  the 
leamer/participant).  Defining  goals  in  terms  of  what  the  learner  will  actually  do  is  also 
critical  to  this  process  (Price  actually  refines  this  prescription  to  stating  goals  in  an  active 
voice  versus  passive  voice  -  “The  learner  will  build  a  nuclear  reactor,”  versus,  “This 
lesson  is  about  building  nuclear  reactors”).  Objectives,  simply  stated,  are  used  to  indicate 
the  performance  of  the  learner. 

The  work  of  Roth  and  Hefley  (1993)  considers  the  technical  perspectives  of  many 
investigations  in  intelligent  multimedia  presentation  systems  (IMMS).  They  review  IMMS 
with  regard  to  its  purpose,  key  functional  requirements,  and  architectural  structure.  They 
also  consider  the  nature  of  information  presented  in  various  IMMS  systems.  Roth  and 
Hefley  (1993)  describe  two  approaches  to  IMMS  design.  The  first  is  a  “task-analytic” 
approach  that  attempts  to  model  actions,  perceptions,  and  other  cognitions  on  behalf  of  the 
IMMS  user.  The  second  is  a  “plan-based”  communicative  act  view  of  an  IMMS  and 
emphasizes  the  presenter’s  goals. 

There  are  several  other  applications  within  IMMS  research  that  have  similar 
approaches  to  the  current  study.  For  example,  studies  in  factory  management  (Roth  and 
Mattis,  1990,  Gargan,  Sullivan,  and  Tyler,  1988),  financial  models  (Marks,  1991), 
marketing  analysis  (Anand  and  Kahn,  1992),  project  management  (Roth  and  Hendrickson, 
1991),  and  virtual  worlds  (Feiner,  MacIntyre,  and  Seligmann,  1992). 

Literature  “On-ramp”  to  Study 

In  order  to  better  focus  on  how  previous  studies  of  the  STRATEGEM-2  literature 

applies  to  the  current  research,  the  following  Figures  6  through  9  show  how  each  of  the 
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primary  authors  (Sterman,  Richardson  and  Rohrbaugh,  and  Howie  and  others)  have 
considered  the  various  decision  support  factors  that  are  currently  in  the  in  the  dynamic 
decision-making  literature.  Additionally,  the  figures  will  show  how  the  current  author 
(Bois)  has  expanded  upon  the  many  various  dependent  and  independent  variables  to  be 
researched. 


STRATEGEM-2  Literature  Review 
Dependent  Variables 
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Figure  6  -  STRATEGEM-2  Dependent  Variable  Summary 
*Self-Assessment  has  been  added  by  the  author 
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Figure  7  -  STRATEGEM-2  Independent  Variable  Summary  (Decision-maker  Factors) 


STRATEGEM-2  Literature  Review 
Independent  Variables 


Task  Complexity 

Number  of  Variables 


Interaction  Between  Sub-systems 
Random  Variation 
Misc.  Task  Characteristics 


Time  Delays 
Decision  Effectiveness 
Oscillation 
Positive  Feedback  /  Gains 
Real-time  Simulation 


Figure  8  -  STRATEGEM-2  Independent  Variable  Summary  (Task  Complexity) 
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STRATEGEM-2  Literature  Review 
Independent  Variables 


interface  /  Environments 


Built-in  Decision  Rules  /  Heuristics 
Learning  via  Lagged  Effects 
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Figure  9  -  STRATEGEM-2  Independent  Variable  Summary  (Interfaces  /  Environments) 


Literature  Summary 

From  the  literature,  it  is  recognized  that  STRATEGEM-2  is  a  very  difficult  game  to 
play  because  of  time  delays,  nonlinearities,  and  positive  feedback  loops.  In  determining  the 
reason  (or  reasons)  why  people  do  poorly  in  the  exercise  has  been  the  subject  of  much 
disagreement,  particularly  when  it  comes  to  suggesting  methods  for  making 
improvements. 

What  is  evident  from  the  literature  is  that  first,  the  forms  of  cues  are  important  to  the 
successful  decision  making  faculties  of  participants  (Richardson  and  Rohrbaugh,  1990; 
Vicente,  1996;  Howie  et  al.,  2000).  Second,  information  presented  to  participants  can  be 
enhanced  through  the  interface  being  used  (Richardson  and  Rohrbaugh,  1990;  Vicente, 
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1996;  Howie  et  al.,  2000).  Third,  participant  knowledge  is  related  to  participant 
performance  (Howie  et  al,  2000).  Fourth,  using  an  interactive  on-screen  tutorial  may 
improve  an  individual’s  knowledge  and  perception  of  the  micro-economy  that  in  turn  may 
improve  game  performance.  And,  finally,  a  self-assessment  of  “effort”  may  have  bearing 
on  the  results  observed. 

Based  upon  the  discoveries  of  the  literature  review,  this  study  turns  toward 
reaccomplishing  major  portions  of  studies  already  undertaken.  Specifically,  by  those  of 
Richardson  and  Rohrbaugh  (1990),  and  Howie  and  others  (2000). 
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Chapter  4 


METHOD  OF  STUDY 


Overview 

The  Richardson  and  Rohrbaugh  (1990)  study,  along  with  Howie  and  others  (2000),  had 
very  sound  theoretical  foundations  in  challenging  a  portion  of  the  misperception  of 
feedback  hypothesis.  Both  studies  attempt  to  fill  gaps  that  are  perceived  to  be  unexplained 
in  the  misperception  of  feedback  hypothesis.  Their  hypotheses,  whether  implicit 
(Richardson  and  Rohrbaugh,  1990),  or  explicit  (Howie  et  al.,  (2000),  stated  that  players  in 
the  STRATEGEM-2  game  do  not  have  perfect  knowledge  of  their  environment,  nor  does 
the  environment  display  perfect  information.  Sterman  (1989a)  would  argue  that  because 
the  participant  can  view  a  graphie  screen  at  anytime  durmg  the  experiment,  they  could 
obtain  immediate  outeome  feedback  of  what  has  been  occurring  in  the  dynamics  of  the 
game.  Richardson  and  Rohrbaugh  (1990)  provided  the  exact  same  outcome  feedback  as 
well  as  provided  current  depreeiation  and  shortfall  information  on  the  game  board.  Howie 
and  others  (2000),  out-doing  their  predecessors,  provided  all  this  information  on  a  single 
game-screen. 

However,  Howie  and  others  (2000)  explicitly  brought  up  a  very  important  facet  of  the 
misperception  of  feedback  hypothesis  worthy  of  further  consideration  -  the  finding  that 
players,  before  and  after  the  game,  could  not  demonstrate  “perfect  knowledge”  of  the 
game.  Richardson  and  Rohrbaugh  (1990)  also  grappled  with  this  aspect  of  Sterman’s 
(1989a)  work  by  asking:  “How  would  players  perform  if  the  computer  screen  direetly 
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provided  them  with  the  cues  appropriate  for  the  task?  What  effect  would  different  forms  of 
cue  presentation  have  on  cognitive  learning?  These  questions  are  important  because  they 
may  reveal  an  alternative  explanation  for  the  misperceptions  and  dysfunctional  behaviors 
formd  by  Sterman  (1989a)  (Richardson  &  Rohrbaugh,  1990,  pg.  464).” 

The  fact  that  Richardson  and  Rohrbaugh  (1990),  and  Howie  and  others  (2000),  amved 
at  mixed  results  (however,  encouraging),  leaves  the  issues  of  “perfect  knowledge,”  modem 
computer  interfaces,  feedforward  cues,  and  the  cognitive  learning  processes  to  be 
unresolved.  Therefore,  the  current  study  retested  the  Richardson  and  Rohrbaugh  (1990) 
precepts.  Additionally,  the  Howie  and  others  (2000),  computer  interface  was  used  along 
with  the  concept  of  testing  participant  knowledge. 

Another  issue  that  has  been  brought  to  the  attention  of  the  researcher  is  by  Rohrbaugh. 
It  refers  to  the  “setup”  of  the  experiment  to  the  participants.  Too  often,  according  to 
Rohrbaugh,  researchers  preoccupy  themselves  with  measuring  the  many  dynamics  of  their 
experiments.  They  bring  in  human  subjects,  give  them  something  to  read,  and  then  move 
into  the  experiment  without  ever  considering  whether  the  setup  may  have  had  an  impact  on 
the  subject’s  performance.  The  concept  of  the  setup  in  the  STATEGEM-2  game  can 
possibly  have  significant  feedforward  effect.  Hsiao  (1999)  found  a  study  where  this 
procedure  was  introduced  as  a  distinct  form  of  measurement.  “Sengupta  and  Abdel-Hamid 
(1993)  base  their  research  design  on  the  theory  of  information  feedback  and  provide 
subjects  with  three  types  of  computer  information  feedback:  outcome  feedback,  cognitive 
feedback,  and  feedforward.  Outcome  feedback  indicates  online  numerical  reports  for 
important  state  variables  of  the  software  project  task.  Subjects  receiving  cognitive 
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feedback  have  access  to  online  time  plots  containing  the  patterns  of  relevant  variables  and 
a  tabular  summary  of  these  cues.  Whereas  outcome  and  cognitive  feedback  are  always 
available  on  computer  screens,  feedforward  is  conveyed  by  an  hour-long  training  session 
prior  to  the  task  (Hsiao,  1999,  pg.  27).” 

Reading  the  instructions  to  STRATEGEM-2,  used  by  the  three  main  studies  identified 
in  this  paper,  many  questions  remained  rmanswered.  Therefore,  improvements  were 
attempted  to  the  setup  of  the  experiment  that  can  transfer  the  dynamics  of  the  game  into 
meaningful  knowledge  that  participants  could  better  grasp  and  understand. 

Furthermore,  the  sample  size  has  been  increased  up  to  seven-fold  fi'om  what  the 
previous  rese^ch  populations  have  been.  For  example,  the  Richardson  and  Rohrbaugh 
(1990)  study  had  18  participants  divided  into  3  different  conditions  of  6  people  each. 
Howie  and  others  (2000)  had  20  participants  divided  evenly  into  2  treatment  groups.  The 
fact  that  neither  study  was  able  to  produce  meaningful  results  may  be  attributable  to  small 
sample  sizes.  The  current  study  had  a  useful  survey  sample  of  1 38  participants. 

Design  Proposal  and  Matrix 

The  following  research  proposal  and  matrix  was  used: 

1 .  Used  the  old  Sterman  instructions  in  the  control  conditions  presented  by  Howie  and 
others  (2000),  (Appendix  D) 

2.  Developed  an  on-screen  tutorial  to  train  game  participants  (Appendix  E) 

3.  Used  the  Howie  STRATEGEM-2  interface  (Appendix  F) 

4.  Tested  game  knowledge  among  the  participants  following  train-up  (Appendix  G) 
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5.  Surveyed  the  participants  to  determine  their  level  of  effort  at  the  end  of  the 
experiment  (Appendix  H) 

6.  Performed  a  practice  trial,  and  then  two  scored  trials  where  orders  to  the  goods 
sector  remained  the  same  (a  single  step  increase  in  year  foiu)  for  each  trial 

7.  Enrolled  1 50  volunteer  participants 
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8.  Randomized  participants  into  4  treatments  and  conditions  as  follows: 


a.  Receives  an  on-screen  train-up  of  the  Sterman  instructions  (presented  by 
Howie  et  al.,  2000),  a  practice  trial,  a  knowledge  survey,  Q&A,  and  two 
measured  trials. 

b.  Receives  an  on-screen  train-up  of  the  Sterman  instructions  (presented  by 
Howie  et  al.,  2000),  a  practice  trial,  a  knowledge  survey,  Q&A,  the 
Richardson  and  Rohrbaugh  decision  rule,  and  two  measured  trials 

c.  Receives  new  on-screen  tutorial  (Bois  instructions),  a  practice  trial,  a 
knowledge  survey,  Q&A,  and  two  measured  trials 

d.  Receives  new  on-screen  tutorial  (Bois  instmctions),  a  practice  trial,  a 
knowledge  survey,  Q&A,  a  practice  trial,  the  Richardson  and  Rohrbaugh 
decision  rule,  and  two  measured  trials 

The  above  treatments  and  conditions  are  further  explained  by  the  following 
2x2  matrix  shown  in  Figure  10  (below).  Along  the  vertical  axis,  there  are  two  treatments 
that  received  (hypothesized)  inadequate/adequate  training  (the  original  Sterman 
instructions,  or  better,  not  receiving  the  Bois  Instructions,  and  receiving  the  Bois 
Instructions).  The  horizontal  axis  has  two  treatments  that  received  (hypothesized)  non¬ 
decision/decision  support  (no  Richardson  and  Rohrbaugh  rule  and  receiving  the 
Richardson  and  Rohrbaugh  Rule).  Four  conditions  are  created  from  the  combinations 
created  by  mixing  different  levels  of  decision  support  and  game  instructions. 
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Figure  10  -  Human  Subject  Random  Group  Assignments 


Data 

The  “final  score”  produced  by  the  simulation  game  was  the  main  data  point  captured  in 
this  research.  The  score  represented  the  subject’s  ability  to  manipulate  capital  sector  orders 
in  order  to  minimize  overall  backlog  orders  against  the  simulation’s  presentation  of  supply 
and  demand,  as  well  as  minimizing  overproduction  of  capital  and  goods  sector  orders.  The 
score  was  determined  by  the  average  absolute  deviation  between  the  supply  and  demand 
for  capital  over  the  length  of  the  game.  The  ultimate  goal  for  the  participant  was  to 
minimize  his  or  her  score  -  the  smaller  the  score,  the  better  the  performance. 

Additional  data  points  collected  included:  The  scores  from  the  participant  knowledge 
surveys  as  tested  by  Howie  and  others  (2000),  (Appendix  G),  and  the  determination  of 
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participant  level  of  effort  from  a  self-assessment  perspective  (Appendix  H).  Additionally, 
an  abundance  of  demographic  information  was  collected  and  analyzed. 

Sample  and  Subjects 

Although  the  planned  research  will  make  inferences  from  the  sample  to  the  greater 
population,  the  researcher  used  a  non-probability/convenience  sample  of  human  subjects. 
Specifically,  participants  were  drawn  from  graduate/imdergraduate  students  enrolled  in  the 
public  administration,  information  science,  business  administration,  finance,  and  marketing 
programs  at  the  State  University  of  New  York  at  Albany.  Student  participants  were  offered 
a  substitute  option  for  other  required  course  requirements  in  order  to  generate  interest  in 
the  experiment.  In  total,  54  graduate  and  96  undergraduate  students  elected  to  participate  in 
this  research. 

Clearly,  this  sample  is  convenient  to  the  researcher,  yet  it  is  also  purposive  in  nature  - 
these  students  where  chosen  because  of  who  they  are,  and  for  the  positions  for  which  they 
are  professionally  preparing  themselves.  These  students  are  getting  ready  for  careers  that 
will  involve  decision  making  that  will  be  rooted  in  complex  systems. 

All  subjects  were  recruited  on  a  voluntary  basis  and  did  so  without  receiving  any 
stipend.  The  participants  were  randomly  placed  into  the  treatments  and  conditions  shown 
in  Figure  10  above.  Because  the  actual  experiment  was  designed  to  last  about  two  hours, 
several  experiment  periods  were  planned  to  accommodate  the  many  and  various  schedules 
of  the  proposed  participants.  There  were  morning,  mid-day,  afternoon,  and  evening 
sessions. 
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Variables  -  Measures 


The  dependent  variables  measured  in  this  experiment  include  the  following:  Score 
received  on  Trial  1,  Score  received  on  Trial  2,  Mean  average  score  for  both  trials,  the 
change  in  scores  between  the  first  and  second  trials  (obtained  by  subtracting  Trial  2  from 
Trial  1),  and  the  self-assessed  level  of  effort.  Independent  variables  included  the  game 
instruction  setup,  decision  support,  game  knowledge,  and  demographic  information. 
Dependent  Variable 

During  the  experiment,  the  dependent  variables  were  the  scores  received  in  the  first 
and  second  trials,  the  mean  average  of  both  trials,  the  change  in  scores  between  the  two 
trials  (as  a  reminder,  the  lower  the  score,  the  better  the  performance  for  a  given  trial),  and 
the  self-assessed  level  of  effort.  The  scores  indicate  the  participant’s  ability  to  ferret  out  the 
important  factors  in  the  decision-making  process  within  the  dynamic  system. 
Operationalization  of  this  variable  was  derived  through  the  actual  decision  process 
required  by  the  participant  to  manipulate  the  computer  simulation.  For  each  of  the  trials  in 
the  experiment,  the  participant  was  faced  with  a  total  of  35  decision  frames  that  spanned  a 
total  of  70  years.  As  the  participant  worked  through  the  several  decision  frames,  his  or  her 
individual  decision  scores  were  accumulated  into  a  final  score. 

The  participant  level  of  effort  was  surveyed  via  a  self-assessment  (Appendix  H).  This 
dependent  variable  has  three  subsets:  self-assessment  of  individual  interest  in  the  research, 
task  understanding,  and  performance.  These  subset  variables  were  operationalized  through 
the  various  statements  found  in  the  survey  instrument  (Appendbc  H). 
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Following  is  the  assignment  of  statements  to  each  of  the  three  subset  variables: 


Variable 

1.  Self-assessment  of  performance: 

2.  Self-assessment  of  research  interest: 

3.  Self-assessment  of  task  imderstanding: 


Survey  Statement  Number 
3, 7, 8, 12 
4,6, 10, 11 
1,2, 5,9 


The  questions  are  designed  to  get  the  participants  to  accurately  document  their 
perceptions  about  their  own  actions  during  the  experiment.  It  was  presumed  that  the 
variables,  when  analyzed,  would  reflect  on  whether  they  had  any  significant  predictability 
upon  the  dependent  variables.  The  survey  instmment  is  based  upon  a  Likert-type  scale.  It 
is  used  to  measure  the  internal  states  of  the  subjects  (such  as  attitudes,  emotions,  and 
orientations)  (Bernard,  2000).  The  design  of  each  question  in  the  instrument  was  used  to 
measure  whether  the  participants  fully  employed  themselves  during  the  experiment. 
Independent  Variables 

Independent  variables  included  the  game  instruction  setup,  decision  support,  game 
knowledge,  and  demographic  information.  Operationalization  of  these  variables  was  as 
follows:  Game  instruction  setup  was  derived  from  subjects  receiving  either  the  original 
Sterman  instructions  (Appendix  D),  or  the  newly  devised  Bois  instructions  (Appendix  E). 

Decision  support  occurred  in  two  forms.  In  the  first,  participants  received,  or  did  not 
receive,  the  Richardson  and  Rohrbaugh  decision  rule.  In  order  to  produce  this  part  of  the 
experiment,  a  card  with  specific  information  about  the  decision  rule  was  provided  to  those 
participants  destined  to  receive  decision  support  (see  Figure  1 1  below). 
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— Front  Side — 

As  the  manager  of  the  STRATEGEM-2  economy,  you  have  taken  it  upon  yourself  to  hire  a  very 
reputable  economic  consultant  to  assist  you  with  your  decisions.  This  person  has  determined  that  if  you 
are  to  follow  the  formula  in  the  blue  box  on  the  reverse  side  of  this  card,  you  will  most  likely  receive 
an  outstanding  score  for  the  game.  You  are  reminded  by  this  professional  that  although  you  are  not 
required  to  heed  the  advice  given,  you  must  remain  patient  and  diligent  with  using  the  formula. 

Example  on  using  the  decision  aide  in  year  zero  of  the  game: 

1 .  Take  the  current  depreciation  of  50  units  and  multiply  it  times  2  (for  100). 

2.  Add  to  that  the  shortfall*  (currently  0)  and  divide  by  2  (which  equals  0). 

3.  Then  subtract  the  current  capital  backlog  {not  total  backlog)  of  50. 

4.  This  produces  an  order  of  50  capital  units  for  Year  0 

*  Shortfall  =  (total  backlogs  -  current  capacity). 

If  this  figure  computes  to  less  than  zero,  use  zero. 


— Back  Side — 

—  To  Order  — 

1.  Plan  in  advance  to  replace  depreciation  loss 

(DEPRECIATION  x  2)  =  _ 

2.  Shortfall;  Reconcile  total  backlogs  with  current  capacity 

add  (SHORTFALL  -  2)  =  + _ 

3.  Adjust  for  prior  orders  not  yet  filled 

subtract  (CAPITAL  BACKLOG)  =  - _ 

Total  Orders  =  _ 


Shortfall  =  (total  backlogs  -  current  capacity). 
If  this  figure  computes  to  less  than  zero,  use  zero. 


Figure  1 1  -  Richardson  and  Rohrbaugh  Decision  Rule  Input  Card 
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In  the  second  form  of  decision  support,  participants  received,  or  did  not  receive,  the 
Bois  instruction  (Appendix  E).  This  instruction  was  designed  as  an  on-screen  tutorial  and 
had  two  learning  inducement  objectives  in  mind  during  development,  1)  to  get  participants 
to  understand  how  STRATEGEM-2  is  played,  the  different  features  of  the  game  board, 
and  what  information  is  being  conveyed  to  the  participant  from  the  various  features  of  the 
interface,  and  2)  to  get  subjects  to  understand  the  concept  of  “equilibrium”  within  the  game 
dynamics.  The  tutorial  was  set-up  as  a  linear  program  to  introduce  the  various  teaching 
elements  and  included  a  branching  design  as  each  successive  page  of  the  tutorial  unfolded 
for  the  participant.  Additionally,  criterion  frames  were  used  to  examine/test  participant 
knowledge  of  the  equilibrium  concept  and  provided  direct  feedback  in  order  to  assist  in  the 
learning  process.  As  a  final  note,  close  attention  was  paid  to  the  passage  lengths  of  each  of 
the  tutorial’s  pages  so  not  to  overtax  subject  attention  spans. 

Game  knowledge  was  tested  by  adapting  the  Howie  and  others  (2000)  knowledge 
survey  (Appendix  G).  Scoring  of  this  survey  was  based  upon  the  number  of  correct 
responses  on  a  0  to  100  percentage  scale. 

Regarding  demographics,  operationalization  of  this  variable  included  participant: 
gender,  age,  graduate  status,  years  of  professional  experience,  total  time  on  task,  and  test 
scores  (knowledge  smvey). 

Experiment  Setup 

For  all  conditions  surveyed,  the  setup  of  the  experiment  included  either  the  original 

Sterman  instructions  (Appendix  D)  or  the  Bois  instructions  (Appendix  E)  along  with  an 

overview  of  the  Howie  STRATEGEM-2  interface  (see  Figure  12  below).  Additionally,  the 
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conditions  either  included,  or  did  not  include,  the  Richardson  and  Rohrbaugh  decision  rule. 
After  a  train-up  session  was  conducted,  a  practice  trial  of  the  game  was  played  by  each 
subject  followed  by  the  knowledge  survey  and  a  question  and  answer  session. 


Figure  12  -  Howie  STRATEGEM-2  Interface 

Data  Collection  Procedure 

In  all,  150  graduate  and  undergraduate  students  at  the  University  at  Albany  volunteered 
to  participate  in  the  experiment.  Each  participant  was  randomly  assigned  to  the  various 
treatments  of  the  experiment. 

The  first  step  of  data  collection  process  was  to  collect  knowledge  survey  (test  score) 
data.  It  measured  the  depth  of  participant  knowledge  of  the  STRATEGEM-2  system.  This 
survey,  or  test  (Appendix  G),  was  administered  following  the  train-ups  of  all  participants. 
The  survey  was  scored  by  the  number  of  correct  answers  based  on  a  0  to  100  percentage 
scale. 
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After  the  setups  for  all  participants  were  completed,  along  with  their  knowledge 
surveys,  each  group  underwent  two  scored  trials  of  STRATEGEM-2  using  the  Howie 
interface.  This  phase  of  the  experiment  collected  individual  participant  data  for  the  Trial  1 
and  Trial  2  final  scores,  the  mean  average  score  for  the  two  trials,  and  the  change  in  scores 
by  subtracting  Trial  2  fi'om  the  Trial  1  score. 

Following  game  play,  all  participants  were  administered  the  post-experiment  written 
survey  as  follows:  Upon  completion  of  the  gaming  simulation,  participants  were  presented 
with  the  self-assessment  survey  instrument  and  briefed  about  its  contents,  as  well  as  about 
researcher  expectations. 

Data  Analysis 

All  statistical  analyses  for  this  research  were  performed  using  the  Statistical  Package 
for  the  Social  Sciences  software  (SPSS).  The  data  analyses  included  simple  descriptive 
statistics  that  were  used  to  capture  the  broad  spectrum  of  data  points  among  the 
participants. 

The  main  analysis  performed  was  a  3-way  analysis  of  variance.  It  was  used  for 
comparison  of  the  instruction  set  (receives  Sterman  instructions  or  receives  the  Bois 
instmctions)  put  against  the  decision  support  rule  (receives  or  does  not  receive  the 
Richardson  and  Rohrbaugh  decision  mle),  and  further  compared  with  a  measure  of  self- 
reported  motivation.  The  analysis  was  used  to  determine  the  main  effect  of  either  the  Bois 
instmction,  the  Richardson  and  Rohrbaugh  decision  rule,  and  motivation  level  upon 
participant  performance. 
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Data  Reduction 


When  considering  the  entire  data  set  after  all  collection  had  been  completed,  it  became 
obvious  that  some  scores  obtained  in  the  two  recorded  trials  were  so  high,  that  some  form 
of  reduction,  or  elimination  of  cases,  would  be  required  in  order  to  better  capture  the  tme 
performance  of  the  body  of  participants,  and  attempt  to  reduce  or  eliminate  problems 
associated  with  regression  to  the  mean.  For  example,  over  two  thirds  of  all  participants 
scored  less  than  1,000  points  for  either  Trial  1  or  Trial  2  (Reminder:  The  lower  the  score 
the  better  the  performance.  The  Sterman  optimal  score  for  the  game  is  19,  and  the 
Richardson  and  Rohrbaugh  decision  mle  produces  and  optimal  score  of  67).  Additionally, 
three  subjects  scored  in  excess  of  10,000  points  in  both  Trials  1  and  2. 

The  researcher  has  determined  that  individuals  receiving  very  high  scores  possibly  did 
not  imderstand  the  task,  or  they  failed  to  grasp  the  requirements  of  the  instructions.  In 
order  to  set  some  sort  of  demarcation,  any  case  with  a  Trial  1  or  Trial  2  outlier  in  excess  of 
4,000  points  was  eliminated  fi-om  the  data  set.  Therefore,  12  cases  were  eliminated  from 
the  original  150,  reducing  the  total  N  to  138,  or  by  8  percent. 
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Data  Conversion 


In  order  to  better  visualize  the  score  data  obtained  during  the  experiment,  Figure  13 
shows  boxplots  for  Trial  1  (Tl),  Trial  2  (T2),  and  the  two-trial  average  (TA)  for  the  four 
conditions  generated  from  the  various  treatments.  Additionally,  it  demonstrates  the 
existence  of  several  mild  and  extreme  outliers  of  the  raw  scores. 


Figure  13  -  Raw  Score  Boxplots  by  Group 


As  can  be  seen  in  this  view  for  data  depiction,  the  scores  produced  for  the  participants 
for  Trial  1,  Trial  2,  and  the  two-trial  average  had  large  ranges,  coupled  with  their  large 
standard  deviations  (some  even  larger  than  their  means),  a  method  to  compress  the  data 
was  searched  for  that  could  effectively  convert  the  data  in  hopes  of  reducing  the  large  size 
of  the  standard  deviations  and  reducing  the  number  of  outliers.  Therefore,  transformations 
of  the  data  that  were  attempted  included  square/cube  root  conversions  and  logarithmic 
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conversions.  Using  a  base  10  logarithmic  conversion  of  the  scores  proved  to  provide  the 
best  compression  of  the  data  and  elimination  of  outliers,  while  at  the  same  time, 
maintaining  the  integrity  of  how  the  data  relates  to  each  other  among  the  various  treatment 
groups.  Figure  14  (below)  demonstrates  the  improvements  made  by  converting  the  scores 
to  their  base  10  logarithmic  equivalents. 


Figure  14  -  Base  10  Logarithmic  Transformation  of  Scores  by  Group 


Data  Description 

Descriptive  information  for  all  variables  (a  total  of  54  data  points  were  collected  for 
each  participant)  is  not  shown  in  this  section.  Only  pertinent  variables  that  may  have  some 
bearing  on  the  research  are  discussed  (for  a  complete  list  of  all  variables  collected  in  the 
research,  refer  to  Appendix  I).  Descriptive  information  for  pertinent  variables,  along  with 
their  SPSS  codes,  is  as  follows: 
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1 .  Gender:  Male  =  1 ,  F emale  =  2 

2.  Age:  Age  in  whole  years 

3.  Grad:  SUNY  status:  1  =  Undergraduate  student,  2  =  Graduate  student 

4.  Exp:  Years  professional  experience 

5.  TT:  Total  time  used  to  complete  the  experiment 

6.  LoglOTl^^:  Base  10  logarithmic  conversion  of  the  T1  score 

7.  LoglOT2^:  Base  10  logarithmic  conversion  of  the  T2  score 

8.  LoglOTA^^:  Base  10  logarithmic  conversion  of  the  TA  score 

9.  DeW^:  Change  in  scores  between  trials  (LoglOTl  -  LoglOT2) 

10.  TS:  Test  score  (knowledge  survey  result) 

11.SA3:  Self-assessment  survey  question  3  (1  to  5  scale  -  1  represents  strongly 
disagrees,  5  represents  strongly  agrees) 

12.  SA3FrVE^^:  Identifies  participants  that  scored  SA3  with  a  “5” 

Motivation  Factor 

At  this  point,  special  emphasis  needs  to  be  made  regarding  how  the  level  of  effort  was 
operationalized  during  the  data  analysis  process.  Recapping  the  initiative  in  this  area, 
Hsiao  (1999)  discovered  only  three  methods  of  measuring  “level  of  effort”  on  behalf  of 
participants  in  a  dynamic  decision-making  (DDM)  study.  They  are:  First,  is  the  amount  of 
decision  time  (how  long  does  it  take  to  make  a  decision).  Second,  is  the  amount  of 
information  use  for  specific  information  items  (is  the  participant  using  the  information 


”  These  variables  were  not  “collected,”  rather,  they  were  computed  within  SPSS. 
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provided  in  the  experiment).  Third,  is  the  amount  of  discussion  among  participants  (do 
they  seek  each  other’s  help  when  allowed  by  the  experiment). 

This  researcher,  interested  in  this  aspect  of  DDM,  posits  that  if  human  subjects  really 
tried  hard,  they  would  perform  well  with  respect  to  the  various  treatments  they  are  exposed 
to  in  the  current  experiment.  The  idea  was  to  administer  a  post-experiment  self-assessment 
survey  where  subjects  would  be  able  to  self-identify:  1)  how  hard  they  were  trying,  2)  then- 
knowledge  of  the  game,  and  3)  their  interest  in  the  research  project. 

After  several  analyses,  it  was  discovered  fi-om  the  subjects’  self-assessment  survey  that 
their  “task  knowledge”  and/or  their  “interest  in  the  research”  were  not  good  predictors  of 
their  effort.  However,  the  statements  regarding  their  performance  in  the  self-assessment 
survey  may  have  been  somewhat  ambiguous  —  except  for  one  statement.  The  variable,  SA3 
(self-assessment  survey  item  #3),  stated:  “I  did  my  best  in  performing  during  this 
experiment;”  the  position  of  this  statement  is  establishes  that  if  someone  was  really  trying 
hard,  he  or  she  would  give  this  a  top  rating  of  “5”  (meaning  that  they  strongly  agree  with 
the  statement).  It  is  believed  that  this  one  measure  alone  can  identify  a  subject  who  was 
“motivated.”  All  others  ranking  this  statement  less  than  “5”  is  considered  to  be 
unmotivated,  or  at  least,  not  as  motivated  as  the  researcher  would  like  them  to  be. 

Extending  the  logic  of  motivated  vs.  unmotivated,  the  boxplots  below  in 
Figure  15  show  a  marked  difference  fi-om  the  boxplots  shown  earlier  for  all  cases.  Here, 
the  motivated  individuals  by  group  have  been  separated  fiom  those  who  are  unmotivated. 
Clearly,  fiom  a  descriptive  point  of  view,  the  differences  in  performance  between  those 
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who  are  motivated  and  those  who  are  not  appear  to  be  noteworthy,  and  warrant  further 
investigation. 


Figure  15  -  Performance  Comparisons  s  of  Motivateds  vs.  with  Unmotivateds 


Limits  of  Research  Design 

The  greatest  threat  to  the  research  design  is  that  of  being  able  to  provide  alternative 
explanations  of  the  results.  Where  evidence  was  found  in  support  of  the  research 
hypotheses,  the  researcher  must  carefully  consider  what  other  explanations  there  would  be 
as  to  why  the  results  turned  out  the  way  they  did  (and  possibly  that  the  results  were  not  a 
consequence  of  the  experimental  intervention). 

Where  the  research  indicates  a  positive  relationship  between  the  intervention  and  the 
overall  participant  performance,  could  this  be  explained  by  one,  or  some,  of  the  kinds  of 
confounds  suggested  by  Bernard  (2000).  They  are: 

1.  History 

2.  Maturation 
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3.  Testing 

4.  Instrumentation 

5.  Regression  to  the  mean 

6.  Selection  of  participants 

7.  Mortality 

8.  Diffusion  of  treatments 

For  the  current  experiment,  no  threats  can  be  discerned  that  are  caused  by  history, 
maturation,  instmmentation,  regression  to  the  mean,  mortality,  and  diffusion  treatments. 
Conversely,  testing  is  a  real  threat,  since  participants  were  given  a  practice  trial  along  with 
two  or  three  measured  trials.  It  is  possible  that  performance  improves  over  time  because 
the  subject  has  merely  gotten  used  to  the  simulation.  It  will  be  important  to  analyze  any 
improvements  made  by  the  intervention  groups  with  any  improvements  made  by  the 
control  groups.  The  analysis  of  variance  using  the  change  in  scores  between  the  first  and 
second  trials  was  used  to  ferret  out  testing,  or  iteration,  effects. 

The  selection  of  participants  is  most  likely  the  strongest  threat  to  external  validity 
under  the  current  research  design.  Although  the  researcher  randomly  assigned  participants 
to  various  treatments,  one  must  realize  that  a  convenience  sample  may  still  affect  the 
external  validity  of  the  research  design.  It  is  possible  that  if  the  same  experiment  was  to  be 
conducted  with  a  different  sample  population  (e.g.,  government  bureaucrats,  business 
leaders,  store  clerks,  laborers,  etc.)  the  results  may  be  different.  This  is  even  more  of  a 
factor  where  the  research  provides  evidence  in  support  of  the  hypotheses.  In  this  event. 
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future  studies  are  recommended  to  see  if  replication  of  results  can  be  found  among  other 
or,  more  disparate  subjects. 

In  the  final  analysis,  the  researcher  believes  that  the  current  experimental  design  has  a 
high  level  of  internal  validity.  However,  there  exists  the  liability  of  low  external  validity.  It 
will  be  very  difficult  to  generalize  the  results  of  a  very  controlled  lab  experiment  to 
decision  makers  in  the  real  world.  This  is  compounded  by  the  fact  that  the  experunent 
contains  a  high  level  of  artificiality.  Yet,  where  the  experiment  provides  evidence  to 
support  the  hypotheses,  it  represents  a  stepping-stone  for  future  research  to  take  a  more 
empirical  approach  to  the  overall  questions  raised  in  the  study  because  it  shows  that 
decisions  within  complex  systems  can  be  improved  with  proper  training,  knowledge, 
motivation,  and  proper  focus  on  pertinent  cues.  Therefore,  the  results  will  allow  for 
empirical  interventions  in  natural  settings  whereby  the  decision  makers  could  have  their 
own  work  environments  modeled,  and  then  be  trained  on  which  cues  (along  with  judgment 
functions)  to  apply  to  their  everyday  problem  solving. 

Limitations  related  to  the  experiment  surveys  are  most  closely  related  to  the  fact  that 
the  written  surveys  were  not  the  primary  method  of  data  collection.  It  was  ancillary  to  the 
computer  simulation  experimentation  process.  The  written  surveys  were  used  to  satisfy  the 
researcher  that  the  subject:  a)  was  interested  in  the  research,  b)  understood  the  tasks,  and  c) 
participated  fully  (Appendix  H),  and  to  determine  their  knowledge  of  the  dynamics  of  the 
microeconomy  (Appendix  G). 

Internal  validity  is  defined  by  the  degree  to  which  one  can  be  certain  that  changes  in 
the  dependent  variable  are  caused  by  the  treatment  (Bernard,  2000),  and  that  the  variables 
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are  linked  together  in  a  relationship  (Krathwohl,  1998).  Here,  the  researcher  is  attempting 
to  establish  such  a  relationship  between  participant  self-perceptions  and  participant 
performance.  Granted,  it  may  be  possible  to  associate  poor  self-assessments  with  poor 
simulation  performance  scores;  however,  one  must  consider  the  possibility  of  associating 
high  self-assessments  with  superior  performance.  In  other  words,  subjects  who  are  trying 
hard  should  produce  better  game  scores. 

External  validity  would  come  under  scrutiny  for  the  following:  That  the  self- 
assessment  survey  indicates  that  participants  did  not  give  it  their  best  effort,  or  they  did  not 
imderstand  the  tasks,  or  were  simply  not  interested.  It  is  the  opinion  of  the  researcher  that 
to  make  any  generalizations  of  performance  when  confronted  with  low  self-assessments 
would  be  meaningless. 

Conversely,  the  external  validity  of  the  study  is  enhanced  where  participants  are  shown 
to  have  done  their  best  from  a  self-assessment  perspective.  The  rationale  for  this  inference 
is  that  the  generalization  of  the  treatment  will  not  be  able  to  be  discounted  from  the 
position  that  the  subjects  failed  to  provide  adequate  participation.  In  other  words,  the  final 
simulation  performance  results  will  be  more  important  when  attributed  to  those 
participants  who  try  their  best  in  performing. 

Ethical  Considerations 

Anytime  one  uses  hxunan  subjects  in  a  research  design,  ardent  ethical  considerations 
must  follow.  The  biggest  item  the  researcher  attempted  to  maintain  was  to  treat  subjects 
with  respect.  Not  all  performed  brilliantly;  some  even  performed  poorly,  yet  that  was 
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expected.  However,  they  were  all  treated  the  same  -  with  dignity  and  appreciation  for 
having  volunteered  their  time  and  effort. 

The  issues  of  confidentiality  and  privacy  should  also  not  go  without  mention.  It  was 
critical  for  the  human  subjects  to  know  that  their  performance  in  the  experiment,  and  their 
answers  and  comments  provided  in  the  surveys,  were  completely  private  and  confidential. 
Only  two  people  will  ever  know  the  individual’s  experimental  performance:  the  researcher 
and  the  participant.  To  break  the  trust  established  by  the  researcher  and  human  subject 
would  decrease  the  quality  and  quantity  of  any  future  research.  All  endeavor  has  been 
made  to  maintain  this  trust  between  researcher  and  participant. 
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Chapter  5 


FINDINGS 


Descriptives 

Considering  the  descriptives  statistics  for  all  participants  (Table  2  below),  the  gender 
difference  is  near  evenly  split  (53%  female).  The  age  of  participants  ranged  from  19  to  51, 
but  the  mean  and  standard  deviation  indicate  the  age  spread  was  predominately  young.  A 
similar  skew  occurs  with  experience  -  low  experience.  Graduate  students  made  up  34%  of 
participants  tested.  The  total  time  participants  took  to  undergo  the  experiment  ranged  from 
55  to  173  minutes  (average  time  was  94  minutes).  Test  score  data  (results  of  the 
knowledge  survey)  ranged  from  26  to  91  with  a  mean  of  55  and  was  evenly  distributed 
(see  Figure  16).  Regarding  the  Base  10  Logarithmic  scores  obtained,  the  three  variables 
measured  have  standard  deviations  that  are  very  small  compared  to  their  means.  The  Delta 
(change  from  the  Trial  1  to  Trial  2  scores)  ranged  from  a  -.76  (participants  doing  worse  in 
the  second  trial)  to  1.31  (participants  doing  better  in  the  second  trial)  and  had  a  mean  of  .07 
(indicating  that  the  number  of  participants  who  did  better  in  the  second  round  of  play  was 
larger  than  those  in  the  first).  The  final  two  variables,  SA3  and  SA3Five  simply  show  the 
range  of  motivation  (SA3)  and  that  the  number  of  motivated  participants  (SA3Five) 
represented  51%  of  the  sample  population. 
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Descriptive  Statistics  of  Pertinent  Variables? 


Minimum 

Maximum 

Mean 

Std. 

Deviation 

Gender 

1 

2 

1.53 

.50 

Age 

19 

51 

23.43 

4.81 

Graduate  Level 

1 

2 

1.34 

.48 

Experience 

0 

33 

1.71 

4.10 

Total  Time 

55 

173 

93.55 

21.72 

Test  Score 

26 

91 

54.75 

15.09 

LoglOTl 

1.83 

3.60 

2.73 

.40 

LoglOT2 

1.76 

3.55 

2.66 

.45 

LoglOTA 

1.86 

3.45 

2.74 

.39 

Delta 

-.76 

1.31 

.07 

.41 

SA3 

5 

4.38 

.74 

SA3Five 

0 

1 

.51 

.50 

a.N=  138 


Table  2  -  Descriptives  for  All  Participants 


Figure  16  -  Histogram  of  Test  Scores  (Knowledge  Surveys) 


It  was  important,  however,  to  determine  if  the  participants  were  randomly  distributed 

among  the  four  conditions  established  by  the  method  of  study  (Bois  instruction  vs.  no 
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instruction,  and  Richardson  and  Rohrbaugh  rule  vs.  no  rule).  To  do  so,  a  one-way  analysis 
of  variance  was  conducted  for  each  of  the  variables  in  Table  2  (cross-checked  against  each 
of  the  foxu"  research  conditions)  to  see  if  any  non-random  assignments  could  be  found  as 
significant  (p  <  .05).  The  result  of  this  test  indicated  that  no  variable  was  found  to  have  a 
significant  non-random  assignment.  This  means  that  the  assignment  of  participants  to  the 
various  treatments  was  indeed  statistically  random. 

Research  Hypotheses 

To  facilitate  reader  comprehension  of  the  analyses  in  the  remainder  of  this  chapter,  a 
review  of  the  research  hypotheses  are  presented.  Assuming  there  are  ways  to  improve 
human  performance  in  the  face  of  time-delayed  feedback  dynamics,  the  following 
hypotheses  were  projected  for  this  research  thesis: 

1 .  If  information  and  knowledge  about  a  system  are  better  understood,  participant 
performance  will  improve. 

2.  If  participants  are  provided  with  a  decision  mle  that  focuses  their  attention  on 
proper  cues  and  how  to  weigh  their  importance,  their  performance  will  improve. 

3.  Participants  reporting  greater  effort  during  the  experiment  simulation  will  out¬ 
perform  those  who  do  not. 
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Analysis  of  Variance 

As  a  reminder  to  the  reader,  Figure  17  (below)  is  provided  in  order  to  show  the  specific 
treatments  and  their  associated  conditions. 


Conditions 

and  . 

Treatments 

No 

Rule 

Receives 

R  &  R  Rule 

No 

Bois 

Instruction 

I 

II 

Receives 

Bois 

Instruction 

in 

IV 

Figure  17  -  Human  Subject  Random  Group  Assignments 


The  first  hypothesis,  improving  knowledge  and  information,  is  represented  by  applying 
the  Bois  instmction.  The  second  hypothesis,  focusing  participant  attention  on  proper 
decision  cues  and  weights,  is  identified  by  the  application  of  the  Richardson  and 
Rohrbaugh  Rule.  The  last  hypothesis,  participants  reporting  a  greater  level  of  effort,  is  not 
directly  reflected  in  Figure  17,  however,  it  was  included  as  a  third  factor  in  the  analysis  of 
variance. 
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The  main  effects  observed  in  the  analysis  of  variance  are  shown  in  Figures  18  through 
21  (below).  Four  ANOVAs  were  performed.  They  included  analyses  of  the  first  trial, 
second  trial,  the  two-trial  average,  and  a  delta  (Trial  1  minus  Trial  2).  The  predominant 
trend  that  is  seen  in  the  following  figures  is  that  the  mean  performance  scores  for  the  first 
and  second  trials,  along  with  the  two-trial  average,  show  improvement  when  either  the 
Bois  instruction,  or  Richardson  and  Rohrbaugh  rule,  is  applied.  Additionally,  there  is  a 
pronoimced  improvement  in  scores  on  behalf  of  participants  who  were  assessed  as 
motivated  over  those  who  were  not. 


Figure  18  -  AN OVA  Findings  for  Trial  1 
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3x3  ANOVA  Means  for  LoglOT2 


2.74 
(n  =  68) 


2.58 

(n=70) 


2.70 
(n  =  70) 


2.61 
(n  =  68) 


2.66 

(n  =  138) 


Motivation 

Totals 


Unmotivated  Kin: 


Motivated 


Figure  19  -  ANOVA  Findings  for  Trial  2 


75 


Figure  21  -  AN OVA  Findings  for  Two-Trial  Delta 

The  analysis  in  the  Delta  category  (change  in  performance  from  Trial  1  to  Trial  2) 
yielded  no  worthy  information  —  no  findings  were  found  to  be  significant.  This  is  probably 
due  to  having  57  cases  performing  worse  in  the  second  trial. 
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In  an  attempt  to  better  show  (graphically)  the  results  of  the  four  ANOVAs  (Trial  1, 
Trial  2,  the  Two-Trial  Average,  and  the  Delta  between  trials),  the  following  Figures  22 
through  25,  demonstrate  the  results  of  each  of  the  four  analyses.  What  is  important  to 
remember  is  that  the  circles  represent  participants  not  receiving  the  Bois  instruction,  the 
triangles  represent  the  reception  of  the  Bois  instruction,  and  the  left  aligned  circles  and 
triangles  represent  participants  not  receiving  the  Richardson  and  Rohrbaugh  rule 
(compared  to  those  aligned  on  the  right  side  of  the  chart  -  they  received  the  mle). 
Motivation  is  also  separated  by  color  as  indicated. 


77 


What  is  important  to  discern  in  Figures  22  through  25  is  the  difference  in  performance 
between  the  motivated/unmotivated  subjects.  For  example,  in  Figures  22,  23,  and  24,  the 
unmotivated  subjects  show  little  or  no  improvement  between  those  who  received  the  Bois 
instmction  and  those  who  did  not.  The  same  relationships  can  be  discerned  between  those 
subjects  receiving  the  Richardson  and  Rohrbaugh  mle  to  those  who  did  not.  However, 
what  is  critically  important  to  observe,  is  that  among  motivated  subjects,  the  differences  in 
performance  between  the  those  who  received  the  Bois  instmction  and  those  who  did  not, 
along  with  the  comparison  of  subjects  receiving,  or  not  receiving,  the  Richardson  and 
Rohrbaugh  mle,  all  perform  as  hypothetically  predicted  (this  precludes  observations  in  the 
Delta  category,  which  are  statistically  insignificant). 


79 


The  following  table  shows  the  F-ratios  obtained  in  the  analysis  of  variance.  Although 
the  instruction,  rule,  and  motivation  factors  had  no  significant  bearing  on  the  Delta 
variable,  all  three  main  effects  were  significant  when  the  two-trial  average  was  used  as  the 
dependent  variable.  Motivation  also  had  a  significant  main  effect  in  the  first  and  second 
trails.  Additionally,  the  mle  had  a  significant  main  effect  in  the  second  trial. 


F-Ratio  of  Main  Effects 


LoglOTI 

Log10T2 

Log IOTA 

Delta 

Instruction 

3.11 

2.11 

4.13* 

.01 

Rule 

3.31 

4.72* 

5.05* 

.35 

Motivation 

11.07** 

4.74* 

10.93** 

.64 

*  Sig.  at  the  .05  level 
**  Sig.  at  the  .001  level 


Table  3  -  F-Ratios  of  Main  Effects  for  Instruction,  Rule,  and  Motivation 

Table  4  shows  the  F-ratios  of  the  interactions  between  the  main  effects  of  the  Bois 
instruction,  the  Richardson  and  Rohrbaugh  rule,  and  the  motivation  factors.  All  F-ratios 
were  found  to  be  non-significant. 


F-Ratio  of  Main  Effect  interactions 


LoglOTI 

Log10T2 

LoglOTA 

Delta 

Instruct  *  Rule 

.09 

1.08 

.31 

.64 

Instruct  *  Motivation 

.80 

1.96 

1.25 

.67 

Rule  *  Motivation 

1.41 

2.55 

2.43 

.40 

Instruct  *  Rule  *  Motivation 

.22 

.13 

.11 

.33 

Table  4  -  Interaction  F-Ratios 
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Motivation  Factor  Explained 

In  the  third  hypothesis,  “participants  reporting  greater  effort  will  out-perform  those 
who  do  not,”  two  two-way  analyses  of  variance  were  performed  to  determine  the 
significance  of  the  Bois  instruction  and  the  Richardson  and  Rohrbaugh  decision  rule  with 
those  who  are  motivated,  and  those  who  are  not.  Table  5  shows  the  results  of  these  two 
analyses. 


LoglOTA  Motivation  F>Ratios 

Unmotivated  Motivated 
Instruction  .44  4.80* 

Rule  .25  7.02** 

*  SIg.  at  the  .05  level 

**  Sig.  at  the  .01  level 

Table  5  -  Motivation  F-Ratios\ 

As  a  final  addendum  to  this  section,  another  very  interesting  discovery  was  made  when 
comparing  participant  knowledge  survey  scores  to  their  self-assessed  motivation  levels. 
The  boxplots  below  in  Figure  26  show  a  very  different  level  of  performance  in  test  scores 
(knowledge  survey)  between  the  two  motivation  levels.  Below  the  boxplots  (Table  6)  are 
the  descriptives  for  these  two  levels  of  measurement,  along  with  a  two-tailed  significance 
test  of  their  means.  The  means  differences  between  them  are  not  only  large  (10  points),  but 
their  significance  is  at  the  .0001  level. 
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100 


N=  70  68 

Motivated  Uhmot'vated 


Motivation 

N=138 

Figure  26  -  Test  Scores  by  Motivation 


Motivation  Comparisions 


Std. 

Mean 

Deviation 

Motivated 

59.61 

15.00 

Unmotivated 

49.75 

13.55 

Mean  Difference 

9.86**** 

****Sig.atthe.0001  level 


Table  6  -  Two-Tailed  T-Test  of  Motivation  between  Groups 


The  significant  differences  in  test  scores  may  attribute,  in  some  way,  to  the  increased 
performance  of  simulation  scores  between  the  two  motivation  levels  of  subjects. 

Anecdotal  Observations 

Given  the  statistical  findings  (above),  other  observations  about  the  experiment  must  be 

highlighted.  For  example,  although  it  is  not  quantified,  participants  who  received  the 

Richardson  and  Rohrbaugh  decision  mle  used  it  in  different  ways.  The  researcher  observed 
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that  when  given  the  rule  card,  participants  often  times  would  simply  discard  it.  At  other 
times,  they  would  try  to  perform  the  calculations  prescribed  by  the  card,  only  to  abandon 
the  rule  card  over  time.  Yet,  others  would  follow  the  prescriptions  of  the  rule  to  the  very 
end  of  the  experiment. 

The  “score”  in  the  simulation  was  also  another  area  of  concern.  It  seemed  that  several 
participants  would  focus  too  much  attention  on  this  output  of  the  game  interface.  For 
example,  several  participants  would  preoccupy  themselves  with  trying  to  obtain  a  lower 
score  versus  trying  to  properly  balance  supply  and  demand. 

Depreciation  did  not  seem  to  be  fully  understood  by  many  participants.  It  is  the  ONLY 
means  of  reducing  capital  stock.  In  other  words,  when  current  capacity  was  too  large  for 
the  desired  production,  there  were  several  subjects  who  neglected  to  simply  order  “zero,” 
in  order  to  lower  their  capital  stocks.  Many  participants  failed  to  appreciate  that  during 
times  of  excess  capacity,  depreciation  could  be  \ised  to  assist  them  in  lowering  their 
production  capabilities  in  order  to  try  and  balance  their  supply  with  demand. 

Finally,  as  an  overall  observation,  many  participants  found  the  simulation  very  difficult 
to  understand.  This  is  firom  an  observational  point  of  view  and  could  not  be  corroborated 
with  self-assessment  data. 

Ambiguities 

When  considering  the  questions  posed  during  the  self-assessment  survey  portion  of  the 
experiment,  it  was  discovered  that  those  statements  dealing  with  “task  knowledge”  and 
“research  interest”  where  not  of  any  statistical  value.  However,  statements  dealing  with 
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self-assessment  of  “performance,”  several  ambiguities  were  discovered  that  may  have  led 
participants  to  misunderstanding  what  exactly  was  being  presented.  For  example,  self- 
assessment  statement  #7  says,  “When  provided  with  a  set  of  decision  cues  to  follow,  I 
followed  them  all  the  time.”  The  problem  with  this  statement  is  the  word  “cues.”  What 
does  it  mean?  Is  it  likely  that  the  average  participant  would  not  understand  what  is  being 
stated.  Additionally,  this  statement  was  geared  toward  subjects  who  had  received  the 
Richardson  and  Rohrbaugh  decision  rule,  and/or  subjects  who  received  the  Bois 
instruction.  These  participants  were  pointed  to  specific  elements  of  the  simulation  and  how 
to  react  to  them;  however,  they  were  never  told  that  these  elements  were  “cues.”  This  can 
lead  to  very  inappropriate  understanding  of  the  statement.  Self-assessment  statements  #8 
and  #12  were  found  to  have  similar  ambiguities.  Only  statement  #3  was  discerned  to  be 
unambiguous. 

The  only  other  item  that  can  be  considered  ambiguous  deals  with  the  verbiage  of  the 
Richardson  and  Rohrbaugh  decision  rule  card  that  was  used  for  participants  receiving  the 
rule.  The  card  (see  Figure  11  on  pg  56)  has  two  sides.  On  the  first  side,  the  participant  is 
told  that  they  have  hired  a  reputable  consultant  to  assist  in  STRATEGEM-2  decision 
making.  The  participant  is  reminded  that  they  must  remain  diligent  with  using  the  decision 
formula  presented  by  the  consultant  (which  is  the  Richardson  and  Rohrbaugh  decision 
rule).  A  sample  “work  through”  of  the  rule  is  also  presented  on  the  fi-ont  side  of  the  card. 
On  the  reverse  side  of  the  card  is  a  layout  of  the  formula  that  the  participant  can  use  by 
simply  “plugging  in”  numbers  found  on  the  game  interface.  The  layout  then  provides  a 
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step-by-step  process  whereby  the  participant  then  arrives  at  a  calculated  game  input  -  a 
number  to  be  used  for  capital  goods  orders. 

Several  ambiguities  were  discovered  after  the  fact  that  has  led  the  researcher  to  wonder 
how  effective  the  treatment  of  the  Richardson  and  Rohrbaugh  decision  rule  was  upon 
game  play.  For  example,  on  the  front  and  reverse  side  of  the  card,  the  term  “shortfall”  was 
clarified  for  the  participant.  Directly  below  this  statement  was  added  verbiage  stating: 
this  figure  computes  to  less  than  zero,  use  zero"  An  ambiguity  occurs  because  this  added 
statement  was  meant  to  relate  to  the  computed  final  “total  orders”  produced  by  the 
Richardson  and  Rohrbaugh  decision  rule  and  not  to  the  “shortfall”  amount.  Additionally, 
more  ambiguity  occurs  because  the  rule  does  not  address  when  computations  end  with  a 
value  that  is  not  evenly  divisible  by  10  (because  the  game  interface  rounds  all  values  to 
their  nearest  10). 

The  final  ambiguity  discovered  was  on  the  reverse  side  of  the  card  whereby  the 
shortfall  amount  was  shown  to  be  added  to  the  computed  depreciation  value.  This  is 
correct;  however,  if  the  shortfall  computes  to  a  negative  number,  the  participant  needs  to 
know  that  instead  of  adding,  they  would  now  be  subtracting  the  shortfall  amount  from  the 
computed  depreciation  value.  Appendix  J  contains  an  improved  Richardson  and 
Rohrbaugh  decision  rule  card  for  any  future  research  desiring  to  use  this  approach  in  the 
STRATEGEM-2  game. 

Given  these  findings  regarding  ambiguity  with  the  Richardson  and  Rohrbaugh  decision 
rule,  the  researcher  was  uncertain  as  to  what  their  effects  are  on  the  results  of  those 
treatment  groups  that  were  exposed  to  the  rule.  The  reason  being  is  that  in  the  face  of  the 
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ambiguities,  several  participants  were  able  to  use  the  rule  card  and  achieved  very  low 
scores.  Others  did  not,  but  was  that  a  result  of  the  ambiguities,  or  that  possibly  they  simply 
discarded  the  rule  (as  was  observed  by  the  researcher  as  an  anecdotal  finding),  or  was  it 
that  they  were  simply  not  analytically  inclined  to  fathom  the  directions  proposed  by  the 
Richardson  and  Rohrbaugh  (1990)  formula?  These  questions  cannot  be  fully  resolved. 
However,  as  a  minimum,  mean  scores  of  the  treatment  groups  using  the  Richardson  and 
Rohrbaugh  decision  mle  were  at  a  level  consistent  with  hypothetical  predictions 
(regardless  of  their  statistical  significance).  Findings  for  these  data,  therefore,  will  remain 
as  stated.  As  a  final  statement  for  this  specific  ambiguity,  it  is  felt  that  if  an  improved 
Richardson  and  Rohrbaugh  rule  card  were  used  (Appendix  J),  it  might  result  in  better 
scores  for  those  participants  exposed  to  the  rule. 
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Chapter  6 


CONCLUSIONS 

First  Hypothesis:  The  Impact  of  Knowledge  and  Information 

The  first  hypothesis  in  the  research  postulated;  If  information  and  knowledge  about  a 
system  are  better  understood,  participant  performance  will  improve.  The  control  for  this 
hypothesis  was  represented  by  participants  not  receiving  the  Bois  instruction.  The 
treatment  was  to  introduce  the  Bois  instruction  to  another  set  of  randomly  assigned 
subjects.  The  research  question  associated  with  this  hypothesis  asks:  Can  proper/adequate 
knowledge  and  information  about  the  system  be  taught  to  participants?  The  significant 
F-ratios  found  for  the  mean  average  two-trial  performance  suggest  that  this  may  be  so. 
However,  caution  must  be  exercised.  For  example,  were  these  improved  performance 
scores  due  to  iteration?  Cognitive  style?  Or,  participant  learning  style?  The  answers  to 
these  questions  are  not  known  fi-om  the  current  study  as  these  areas  of  interest  were  not 
measured  during  the  experiment. 

Second  Hypothesis:  The  Impact  of  Decision  Support 

The  second  hypothesis  of  the  study  states:  If  participants  were  provided  with  a  decision 
mle  that  focuses  their  attention  on  proper  cues  and  how  to  weigh  their  importance,  their 
performance  will  improve.  The  control  for  this  hypothesis  was  represented  by  participants 
not  receiving  the  Richardson  and  Rohrbaugh  rule.  The  treatment  was  to  introduce  the 
Richardson  and  Rohrbaugh  rule  to  another  set  of  randomly  assigned  subjects.  The  research 
question  associated  with  this  hypothesis  asks:  Can  participant  performance  be  improved 
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via  decision  cues  and  weights?  As  in  the  first  hypothesis,  the  significant  F-ratio  scores  for 
the  two-trial  average  suggest  that  improvements  to  the  decision-making  process  can  be 
made  through  the  use  of  cues  and  weights.  Again,  did  iteration,  cognitive  style,  or  learning 
style  have  a  factor  in  this  finding?  The  answer  to  this  question  cannot  be  determined  from 
the  current  research  design. 

Third  Hypothesis:  The  impact  of  Level  of  Effort 

The  final  hypothesis  of  the  study  suggested:  Participants  reporting  greater  effort  during 
the  experiment  simulation  will  out-perform  those  who  do  not.  This  hypothesis  is  used  in  an 
attempt  to  answer  the  following  research  question:  Can  a  participant’s  self-assessment  of 
level  of  effort  be  used  to  better  determine  their  own  experiment  performance?  Level  of 
effort  was  operationalized  through  motivational  self-assessment  on  behalf  of  the 
participant.  The  discoveries  made  in  this  area  were  found  to  have  a  noteworthy  impact 
upon  experiment  results. 

The  first  of  these  discoveries  was  found  in  the  significant  F-ratios  for  each  trial  (and 
the  two-trial  average)  of  the  experiment.  These  ratios  suggest  that  subjects  who  really  try 
hard  to  implement  experiment  interventions  consistently  have  a  greater  effect  (improved 
performance)  than  those  who  do  not.  This  is  an  important  finding  in  light  of  the  corpus  of 
the  dynamic  decision-making  literature  for  it  posits  the  question  of  how  important  other 
research  findings  have  been  because  they  have  not  been  filtered/differentiated  for 
motivation  factors. 

The  angle  of  this  specific  portion  of  the  research  is  to  determine  if  there  is  a  masking  of 

the  results  obtained  that  can  somehow  be  peeled  away,  revealing  a  better  understanding  of 
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the  experiment  treatments.  Specifically,  when  the  data  set  was  divided  between  motivated 
and  unmotivated  participants  (as  self-assessed  from  the  viewpoint  that  “they  did  their  best 
while  participating  in  the  experiment),  it  was  found  that  those  who  self-assessed 
themselves  to  be  motivated,  outperformed  those  lacking  full  motivation. 

For  the  motivated  subjects,  significant  F-ratio  results  were  found  for  the  motivated 
participants  versus  the  unmotivated  participants.  Therefore,  it  is  possible  that  lower 
motivation  levels  (those  that  are  not  fully  motivated)  mask  the  intended  treatments  that  are 
designed  to  improve  decision  making  in  the  STRATEGEM-2  environment. 

Using  the  motivation  discriminator  reveals  another  interesting  facet  of  the  research. 
Test  scores  (results  of  the  knowledge  survey)  averaged  about  55  percent  (on  a  0  to  100 
percentage  scale)  for  all  participants.  Yet,  when  considered  independently  between  those 
subjects  that  were  motivated  and  those  that  were  not,  the  mean  scores  were  about  60 
percent  for  motivateds  versus  50  percent  for  unmotivateds.  This  was  a  clear  indication  that 
the  motivational  level  produces  improved  results  upon  performance,  and  it  was  found 
significant  at  the  .0001  level. 

Discussion 

This  dissertation  project  began  with  the  notion  that  the  Sterman  (1989a)  experiment 
with  STRATEGEM-2  may  have  been  flawed  with  respect  to  the  misperception  of 
feedback  hypothesis.  Specifically,  participants  in  the  simulation  performed  poorly  in  light 
of  having  “perfect  knowledge  and  perfect  information”  while  undergoing  the  rigors  of 
play. 
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It  is  the  current  research  initiative  that  the  Sterman  (1989a)  observations  regarding  the 
misperception  of  feedback  hypothesis  remain  accurate  to  some  degree.  This  means  that 
participants  perform  poorly  because  they  fail  to  properly  perceive  the  time  delays  in  the 
system,  and  they  fail  to  understand  the  effect  of  their  decisions  to  their  environment.  These 
elements  of  the  misperception  of  feedback  hypothesis  cannot  be  eliminated  from  current 
findings,  however,  what  cannot  be  corroborated,  is  the  perfect  knowledge  and  information 
premise  made  by  Sterman  (1989a).  For  example,  as  was  performed  in  the  Howie  and 
others  (2000),  knowledge  of  the  simulation  and  system  environment  was  tested  in  the 
current  study.  The  results  in  this  portion  of  the  experiment  once  again  clearly  demonstrate 
that  participants  do  not  possess  perfect  knowledge  of  the  system.  Regarding  whether 
participants  possess  perfect  information  is  also  debatable.  Although  Howie  and  others 
(2000)  were  able  to  demonstrate  how  an  improved  simulation  interface  works  toward 
improving  the  information  about  the  system  to  the  participant  and,  in  turn,  contributes 
toward  better  participant  performance,  one  cannot  say  that  the  information  presented  is 
perfect.  This  facet  was  not  tested  by  Howie  and  others  (2000),  or  by  Sterman  (1989a),  yet 
was  claimed  to  exist  by  Sterman  (1989a).  The  current  study  does  not  profess  that  such  an 
ideal  of  “perfect  information”  exists,  and  it  cannot  be  determined  how  such  a  concept  can 
even  be  measured. 

The  current  research  argues  that  the  notion  of  perfect  knowledge  and  information 
should  no  longer  be  a  part  of  the  misperception  of  feedback  hypothesis.  Rather,  the 
opposite  is  more  probable,  that  perfect  knowledge  and  information  are  not  a  benefit 
enjoyed  by  participants. 
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Given  that  test  subjects  do  not  have  perfect  knowledge  and  information,  it  remains  to 
know  if  they  can  be  taught  to  make  better  decisions  (the  Bois  instruction),  or  can  they  be 
shown  to  make  better  decisions  (the  Richardson  and  Rohrbaugh,  decision  rule).  It  is  felt 
that  this  occurred  on  both  counts  -  particularly  when  participants  where  screened  for  self- 
assessed  motivation  levels.  However,  caution  is  warranted;  for  it  is  unknown  if  the 
improvements  observed  from  the  interventions  were  not  a  result  of  other  issues  that  were 
not  measured  (e.g.  cognition,  learning,  and  iteration).  Given  that  significant  effects  in 
decision  making  were  recorded  for  all  two-trial  average  (LoglOTA)  scenarios,  the  results 
are  still  encouraging  that  either  the  Bois  instruction,  or  Richardson  and  Rohrbaugh 
decision  rule,  were  able  to  assist  decision  makers  improve  their  performance  over  those 
subjects  that  lacked  any  assistance  at  all.  The  misperception  of  feedback  hypothesis 
remains  an  important  barrier  towards  effective  decision  making  in  dynamic  environments; 
however,  this  study  shows  promise  that  decision  makers  can  be  aided  in  improving  their 
decision-making  skill  in  these  environments. 

The  findings  from  the  current  research  indicate  three  important  factors  that  can  be  used 
to  improve  decision-making  support  in  dynamic  environments.  The  first  factor  is 
motivation.  Clearly,  this  factor  produced  significant  results  across  all  levels  of  the 
simulation  and  it  is  important  to  take  notice  of  it.  Decision  support  researchers  and 
consultants  need  to  begin  paying  attention  to  this  factor.  Because  the  lack  of  motivation 
has  a  tendency  to  mask  intended  decision  support  interventions,  it  is  imperative  that 
decision  support  systems,  particularly  those  in  real  world  environments,  consider  ways  to 
motivate  decision  makers  to  become  motivated  at  a  very  high  level.  The  methods  to  do  so 
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are  undetermined  from  the  perspective  of  the  current  research.  However,  they  could 
include  such  things  as:  monetary  reward,  enlisting  decision  makers  to  have  a  greater  “stake 
in  the  outcome,”  and  improved  benefits  (health  coverage,  retirement  benefits,  insurance 
coverage,  compensation  time-off,  improved  workspace,  to  name  a  few).  This  list  is  not 
exhaustive,  yet,  provides  consideration  for  improving  motivation  among  real-world 
decision  makers  who  are  operating  in  dynamic  environments.  For  researchers,  it  represents 
a  possible  list  of  factors  that  can  be  used  to  determine  the  effectiveness  of  improving 
decision-maker  motivation. 

The  second  factor  that  can  be  used  to  improve  decision-maker  performance  is 
instmction.  The  current  research  focused  on  increasing  participant  knowledge  and 
interpreting  information  within  a  simulation  environment.  It  is  posited  that  the  same  can  be 
translated  to  a  real-world  environment.  Researchers  and  consultants  in  this  area  would 
need  to  focus  more  attention  on  trying  to  explain  the  dynamics  of  decision  environments  to 
decision  makers.  For  example,  in  the  current  research,  the  Bois  instruction  focused  very 
heavily  on  explaining  the  concept  of  “equilibrium”  in  the  STRATEGEM-2  environment. 
This  concept  is  not  unique  to  STRATEGEM-2,  but  is  applicable  to  most  dynamic  decision 
environments. 

The  third  factor  that  can  be  used  to  improve  decision-maker  performance  is  mle  based. 
The  Richardson  and  Rohrbaugh  rule  not  only  had  a  significant  effect  on  experiment 
results,  it  is  possible  that  it  provided  great  benefit  to  decision  makers  who  had  most 
difficulty  in  understanding  the  STRATEGEM-2  environment.  It  is  opined  that,  possibly, 
decision  makers  who  are  most  inclined  to  approach  decision  situations  in  an  intuitive 
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(cognitive)  manner  would  benefit  most  from  such  a  decision  rule.  This  is  opposed  to  an 
analytically  inclined  decision  maker  who  would  rely  more  on  his,  or  her,  understanding  of 
the  environment  to  make  better  decisions.  The  effect  of  the  Richardson  and  Rohrbaugh 
mle  cannot  be  truly  appreciated  from  the  current  research  because  cognitive  faculties  were 
not  measured  on  behalf  of  the  participants.  However,  a  measurement  of  cognition  among 
decision  makers  may  possibly  improve  decision  support  interventions.  This  area  is 
imperative  for  future  research  and  should  not  be  overlooked  by  real-world  decision-support 
consultants. 

The  three  factors  identified  as  important  for  decision  support  within  dynamic 
environments  undoubtedly  requires  further  study.  The  STRATEGEM-2  game  is  probably 
a  very  fine  instrument  to  use  to  test  varying  hypotheses  that  will  lead  to  better  decision 
support  in  the  real  world.  To  do  so,  in  the  next  chapter,  a  series  of  recommended  studies  is 
put  forth.  Particular  attention  is  paid  toward  what  the  previous  literature  has  covered, 
toward  what  the  Bois  research  has  posited,  and  toward  what  future  research  should 
include. 

Summary 

In  summary,  the  current  research  concludes  the  following: 

Perfect  information  and  perfect  knowledge  should  no  longer  be  the  premise  of  the 
misperception  of  feedback  hypothesis,  rather,  improvements  can  be  made  with  regard  to 
decision  makers  operating  in  dynamic  environments. 
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The  intervention  of  a  new  instruction,  one  that  teaches  participants  on  the  necessary 
knowledge  to  become  a  better  decision  maker  within  the  dynamic  environment,  has  shown 
to  have  had  a  significant  effect  (p  =  .05)  towards  improving  decision-maker  performance. 
This  was  observed  from  the  main  effect  that  the  instruction  had  upon  the  two-trial  average 
score. 

The  intervention  of  a  decision  support  rule,  one  that  directs  participants  toward  specific 
cues  and  provides  a  weight  for  their  importance,  has  shown  to  have  significant  effects 
(p  =  .05)  toward  improving  decision  maker  performance.  The  rule’s  main  effect  was 
significant  for  the  second  trial  score,  as  well  as  the  two-trial  average. 

Participant  self-assessed  motivation  had  a  significant  effect  (p  =  .05)  on  the  second  trial 
score,  and  it  had  very  significant  effect  (p  =  .001)  on  the  first  trial  and  two-trial  average 
scores.  The  motivation  factor  was  also  found  to  have  a  masking  effect  upon  the  results. 
This  means  that  by  measuring  the  subjects’  motivation  level,  a  truer  overall  picture  of 
performance  can  be  obtained.  This  was  shown  when  a  performance  comparison  was  made 
between  the  motivated  and  unmotivated  subjects.  In  this  case,  the  instruction  and  the  rule 
had  large  and  significant  F-ratios  (p  .05  for  the  instruction,  and  p  =  .01  for  the  rule)  for 
those  participants  who  were  motivated.  For  unmotivated  participants,  the  instruction  and 
the  mle  appeared  to  have  no  effect  at  all  upon  their  performance. 
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Chapter  7 


FUTURE  RESEARCH 

The  results  of  this  study  are  preliminary.  Future  research  is  required  that  will  further 
add  to  the  body  of  knowledge  surrounding  the  current  study’s  findings.  Additional 
exploration  will  also  be  able  to  further  address  key  issues  that  will  better  assist  the 
development  of  improved  decision  support  for  decision  makers  within  dynamic 
systems. 

Future  research  using  the  STRATEGEM-2  simulation  should  address  and/or 
possibly  include  the  following: 

The  effects  of  iteration,  cognitive  styles,  and  learning  styles  should  be  explored  in 
future  studies.  There  exists  a  certain  potential  that  these  criteria  may  unveil  more 
“masks,”  which  may  indicate  a  more  accurate  portrayal  of  research  interventions. 

The  subject  population  requires  expansion.  The  current  research  has  relied 
(conveniently)  on  too  small  of  a  target  population.  Age,  experience,  education, 
background  need  to  be  expanded.  The  current  pool  of  participants  was  very  narrow  in 
scope  in  these  areas. 

In  the  ciirrent  research,  the  use  of  the  STRATEGEM-2  interface  is  taught  solely 
from  a  written  perspective.  It  is  possible  that  the  interface  can  be  “classroom  taught” 
and  would  produce  more  uniform  results  and  improve  upon  subject  knowledge  of  the 
game  dynamics. 
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The  Bois  instruction  can  certainly  be  improved  upon.  As  a  “first  try”  in  producing 
an  instruction  via  on-screen  tutorial  methods,  a  better  computer-assisted  instmction  can 
certainly  be  devised. 

Regarding  the  ambiguities  discovered  with  the  Richardson  and  Rohrbaugh  decision 
mle  card,  using  the  improved  card  (Appendix  J)  is  highly  recommended. 

The  STRATEGEM-2  interface  requires  noteworthy  changes.  The  first  is  the 
elimination  of  the  game  score  being  so  prominently  displayed  in  the  center  of  the 
screen.  Althougli  it  provides  some  sort  of  outcome  feedback,  it  detracts  players  from 
staying  focused  on  more  important  elements  of  the  simulation.  Second,  the  graphs 
showing  historical  feedback  at  the  bottom  of  the  interface  could  most  likely  be 
eliminated  without  any  derogatory  effects.  Third,  there  needs  to  be  a  randomization  of 
the  goods  sector  demand  within  the  simulation.  Currently,  the  goods  sector  takes  a 
single  step  increase  of  50  xmits  in  year  four  during  the  game;  using  a  randomization  of 
increases  or  decreases  would  make  for  a  more  realistic  environment  and  would  also 
increase  the  validity  of  proposed  decision  support  rubrics. 

Finally,  the  greatest  challenge  of  this  type  of  research  is  to  transform  the  dynamics 
of  the  experiment  into  real-world  —  natural  —  settings.  Is  it  possible  to  find  dynamic 
environments  in  the  real  world  that  can  be  used  for  discerning  whether  or  not 
instmctions  or  decision  rules  can  be  of  added  value  within  those  dynamic 
environments?  This  is  the  crux  of  the  current  research  that  is  hoped  will  someday  bare 
important  results. 
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Literature  “Off-ramp”  to  Study 


In  the  following,  Figures  27  through  30,  a  review  of  the  STRATEGEM-2  literature 
is  presented  once  again.  Additionally,  various  arrows  point  to  sub-topics  of  dependent 
and  independent  variables;  they  recommend  continued  research,  potential  for  improved 
research,  and  to  elements  not  recommended  for  future  research  in  a  STRATEGEM-2 
environment. 

STRATEGEM-2  Literature  Review 
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Figure  27  -  STRATEGEM-2  Dependent  Variable  Summary  Recommendations 
*Self-Assessment  has  been  added  by  the  author 


In  Figure  27  (above),  the  author  has  placed  emphasis  on  continued  research  in  the 
area  of  self-assessed  levels  of  effort.  This  area  represents  an  important  discovery  in  the 
current  research  and  should  not  be  overlooked  in  future  experiments  with  hxunan 
subjects  using  the  STRATEGEM-2  simulation.  Additionally,  this  area  may  also  be  of 
great  importance  to  other  studies/experiments  in  dynamic  decision  making. 
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Figure  28  -  STRATEGEM-2  Independent  Variable  Summary  Recommendationspecision-maker  Factors) 
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Figure  29  -  STRATEGEM-2  Independent  Variable  Summary  Recommendations  (Task  Complexity) 


STRATEGEM-2  Literature  Review 
Independent  Variables 
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Figure  30  -  STRATEGEM-2  Independent  Variable  Summary  Recommendations(Interfaces  /  Environment) 
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APPENDIX  A 


Dependent  Variables  for  Dynamic  Decision  Making* 


Categoty 

Measure  and  Studies 

Task 

Pefformance 

Optimizing,  maximizing  or  minimizing,  specified  measures  or  benchmarks 

-  Cost,  the  higher  the  cost,  the  lower  the  performance  (Sterman,  1989a;  Sterman,  1989b; 

Richardson  &  Rohrbaugh,  1990;  Wang,  1994;  Diehl  et  al.,  1995;  Maxwell, 

1995;  Trees  etai.,  1996) 

-  Profit  (Kampmann,  1992;  Bakken,  1993;  Paich  et  al.,  1993;  Yang,  1996;  Young  et 

al.,  1997) 

-  Patients* health  conditions  (Kleinmuntz  et  al.,  1981) 

-  Proportion  of  patients  cured  (Kleinmuntz,  1985;  Kleinmuntz  et  al.,  1987) 

-  Number  (percent)  of  areas  lost  (Brehmer,  1 990;  Brehmer  et  al.,  1991;  Brehmer,  1 995; 

Brehmer  et  al.,  1995) 

-  Difference  (percent  difference)  compared  with  a  benchmark  (Broadbent  et  al.,  1978; 

Mackinnon  et  al.,  1980;  Bakken,  1993) 

—  Number  of  decision  outcomes  better  than  a  benchmark  (Broadbent  et  al,,  1986) 

Reaching  specified  targets 

-  Number  (percent)  of  attempts  within  a  range  of  a  specified  target  (Berry  et  al.,  1984; 

Broadbent  et  al.,  1986;  Berry  et  al.,  1987;  Berry  et  al.,  1988;  Hayes  et  al.,  1988; 
McGeorge  et  al.,  1989;  Sanderson,  1989;  Stanley,  et  al.,  1989) 

—  Number  (percent)  of  attempts  in  correct  directions  to  reach  the  target  (Sanderson,  1989;  Yang, 
1996) 

—  Number  (percent)  of  errors  of  (Erections  to  reach  the  target  (Broadbent  et  al.,  1986;  Berry  et 
al.,  1987) 

Task  systems  behaviors 

-  Number  of  ^sterns  destruction  (Yang,  1 997) 

—  Number  of  appearances  of  an  archet)pe  fixes  that fail**  (Yang,  1997) 

Goals  combining  two  criteria 

-  Market  share  and  cumulative  net  marketing  contribution  (consistent goals)  (Hogarth  et  al., 

1981) 

-  Cost  and  schedule  (confUctinggoals)  (Sengupta  et  al.,  1993) 

Goals  combining  multiple  (greater  than  two)  criteria 

-  A  composite  index  based  on  six  indicators  (Jansson,  1 995) 

*  NOTE:  The  table  in  this  appendix  has  been  adapted/modified  from  {Hsiao,  N.  (1999).  In  search  of  theories  of  dynamic  decision 
making.  A  Bterature  review.  Paper  presented  at  the  International  System  Dynamics  Conference,  Wellington,  New  Zealand.}. 
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Learning 

Mean  scores  of  pre-game  and/or  post-game  questionnaires  on  the  relationships  of 
variables,  including  those  direct  and  crossed  relationships  between  variables 

—  Declarative  task  knowledge  (Broadbent  et  al.,  1978;  Berry  et  al.,  1984;  Broadbent  et 
al.,  1986;  Berry  et  al.,  1987;  Berry  et  al.,  1988;  Hayes  et  al.,  1988;  Sanderson, 
1989;  Bakken,  1993;  Jansson,  1995;  Maxwell,  1995;  Trees  et  al.,  1996;  Howie 
et  al.,  2000) 

Mean  scores  of  pre-game  and/or  post-game  questionnaires  same  as  above 

-  'Procedural task  knowledge  (Hayes,  et  al.,  1988) 

Number  of  correctness  of  mental  models  aligned  with  heuristics  and  goals  set 
forth 

-(Yang,  1996;  Yang,  1997) 

Number  matching  certain  types  mental  models 

-  (Sanderson,  1989) 

Performance  on  transferred  tasks 

-  (Berry  et  al.,  1988;  Hayes  et  al,  1988;  Bakken,  1993;  Wang,  1994) 

Efforts  for 

Amounts  of  decision  time 

Decision 

(Kleinmuntz  et  al.,  1987;  Sanderson,  1989;  Brehmer,  1990;  Brehmer  et  al.,  1991; 

Making 

Sengupta  et  al.,  1993;  Wang,  1994;  Brehmer,  1995;  Brehmer  et  al.,  1995;  Diehl  et 
al.,  1995;  Jansson,  1995;  Maxwell,  1995;  Yang,  1996) 

Amounts  of  information  use  for  specific  information  items 

—  (Brehmer  et  al.,  1991;  Sengupta  et  al.,  1993;  Brehmer  et  al.,  1995;  Jansson,  1995; 
Maxwell,  1995;  Yang,  1996) 

Amounts  of  discussion  among  subjects 

-  (Hogarth  et  al.,  1981) 

Quality  of 

Decision  scope 

Decision- 

Making 

—  Number  of  different  decision  rules  employed  (Wang,  1 994;  Young  et  al.,  1 997) 

Process 

Reliability 

—  Fluctuations  of  decisions  (Sengupta  et  al.,  1993) 

Decision- 

Delegation  of  decision  making 

Making 

—  (Brehmer  et  al.,  1991) 

Architecture 
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Independent  Variables  for  Dynamic  Decision  Making* 


Category 

Conceptual 

Definition 

Measures  and  Studies 

Decision- 

Maker 

Factors 

Cognitive  style 
(ability) 

-  MBTI  QAyers-Bri^s  Type  Indicator)  (Maxwell,  1995;  Trees  et  al.,  1996) 

-  Gregoric  Style  Delineator  (four  mediation  channels)  (Trees  et  al.,  1996) 

-  Gordon's  Cognitive  Style  Indicator  (four types)  (Trees  et  al.,  1996) 

Task  Expertise/ 

Academic 

Training 

-  Whether  subjects  have  task  domain  expertise  in  terms  of  their  academic 

background  (Bakken,  1993) 

-  Whether  subjects  receive  a  2-(kty  session  involving  simulation  of  the  JOBS 

pro^am  (Maxwell,  1995) 

Computing 

skills 

-  Subjects'  self-rating  evaluation  about  their  computer  use  skills  (Trees  et  aL, 

1996) 

Practice  /  task 
experience 

-  Whether  subjects  experience  repeated  trials  (not  explicitly  manipulated) 

(Broadbent  et  al.,  1978;  Kleinmuntz  et  aL,  1987;  Berry  et  al.,  1987; 
Berry  et  al.,  1988;  Stanley  et  al.,  1989;  Brehmer,  1990;  Brehmer  et 
al.,  1991;  Bakken,  1993;  Sengupta  et  al.,  1993;  Paich  et  al.,  1993; 
Wang,  1994;  Diehl  et  al.,  1995) 

-  Amounts  of practice  from  repeated  trials  (Berry  et  al.,  1984;  Broadbent  et 

al.,  1986;  Sanderson,  1989) 

-  Whether  subjects  experience  a  conceptually  similar  task  for  the  next  trial  block 

(Berry  et  al.,  1988) 

Task 

Complexity 

Total  variables 

-  Total  number  of  variables  in  task  systems  (Mackinnon  et  al.,  1 980) 

Interaction  b/w 
subsystems 

-  Whether  interaction  exists  between  variables  or  subsystems  (Mackinnon  et  al., 
1980) 

Random 

variation 

-  Whether  random  variation  exists  at  strategic  points  in  tasks  (Mackinnon  et 
al.,  1980) 

Miscellaneous 

task 

characteristics 

-  Initial  healthy  treatment  risky  and  symptom  diagnosiicity  (Kleinmuntz,  1985) 

"  Treatment  risk  (Appearance  or  strength)  (Kleinmuntz  et  al.,  1987) 

-  luevels  of  price  regime  (Kampmann,  1 992) 

-  Types  of  software  project  (Sengupta  et  al.,  1 993) 

Time  delay  / 
lagged  effects 

-  Tagged  effects  (Broadbent  et  al.,  1978;  Broadbent  et  al,  1986;  Berry  et 

al.,  1988;  Paich  et  al.,  1993) 

-  Time  constants  (Sterman,  1989a;  Sterman,  1989b;  Brehmer,  1990; 

Brehmer  et  al.,  1991;  Kampmann,  1992;  Brehmer,  1995;  Diehl  et 
al.,  1995) 

*  See  NOTE  to  Appendix  A.  Same  source  and  modifications  apply  here. 
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Effectiveness  of 
decisions  on 

outcomes 

-  Treatment  effectiveness  (Kleinmuntz,  1985) 

-  Reducing  stability  by  enlarging  effects  of  a  decision  on  outcomes  (Broadbent 
etal.,  1986) 

-  Effectiveness  of  firefighting  units  (Brehmer  et  al.,  1991) 

Frequency  of 
oscillation 

-  Number  of peaks  of  prices  (Bakken,  1 993) 

Positive 
feedback  and 
gains 

(appearance  or 
strength) 

-  Positive  gains  built  in  the  task  model  (Sterman,  1989a;  Sterman, 

1989b;  Kampmann,  1992;  Diehl  et  ai.,  1995) 

-  Strength  of  ''word  of  mouth  ”  (Paich  et  al.,  1 993) 

-  Number  of  intervals  a  ^stem  falls  in  the  uncontrollable  positive  loops 
(Young  et  al.,  1997) 

Real-time  tasks 

-  Whether  a  task  ^stem  is  clock-driven  or  event-driven  (Brehmer,  1995) 

Decision- 

Making 

Interfaces 

and 

Environments 

Heuristics 
(decision  rules) 
built  in  task 
systems 

-  3  levels:  1 )  arbitrary  consistent,  2)  arbitrary-random,  and  3)  none  (left  for 

human  judgment)  (Hogarth  et  al.,  1981) 

-  3  levels  of  strategies  with  increasing  computational  complexity:  1)generate- 

and-test,  2)  heuristic,  and  3)  EU-bcyesian  (Kleinmuntz  et  al.,  1981) 

-  Random  vs,  schema-driven  strategies,  2  kvels  of  information  acquisition,  2 

levels  of  base-rate  utikr^tion,  3  levels  of  computational  complexity 
(Kleinmuntz,  1985) 

Learning  via 
lagged  effects 

-  Selective-mode  or  unselective  mode  by  varying  lagged  effects  of  decisions 
(Hayes,  et  al.,  1988) 

Heuristics  in¬ 
duced  goal  set¬ 
ting  that  subjects 
receive  through 
verbal  directions 

-  2  types:  1 )  total  assets  goal  (long-term  whole-ystem  goal)  and  2)  total  assets 

and  order  growth  goal  (short-term  subsystem  goal)  (Yang,  1996) 

-  3  types:  pry  (predator  (whole-system)  ratio,  pry! predator  (whole-ystem) 

number,  and  pry  (sub-ystem)  number  (Y^ng,  1997) 

Task  property, 
strategies,  and 
heuristics 
(decision  rules) 
that  subjects 
receive  through 
verbal 
instructions 

-  Training  /  no  training  concerning  task  property  (Berry  et  al.,  1984; 

Berry  et  al.,  1987;  Berry  et  al.,  1988) 

-  3  levels  of  task  property:  1)  no  preliminary  trainings  2)  trained  with 

relationships  of  variables,  and  3)  practicing  each  pair  of  relationships 
separately  (Broadbent  et  al.,  1986) 

-  3  levels  of  expert  transcripts:  1)  no  transcript,  2)  block-by-block  transcript, 

and  3)  whole  transcript  (Stanley  et  al.,  1989) 

-  5  levels  of  instructions:  1)  no  training,  2)  expert  transcript,  3)  memory 

training  4)  rule  construction,  5)  simple  rule  (Stanley  et  al.,  1989) 

-  5  types  of  expert  transcripts:  1)  no  training  2)  initial  blocks,  3)  final 

blocks,  4)  pre-cutpoint  of  peformance,  5)  post-cutpoint  of  performance 
(Stanley  et  al.,  1989) 

-  2  levels  of  instructions:  1 )  ystematic-elaborate:  variables^  relationship,  2) 

goal-planning  detaikd  measures  of  decisions  and  outcomes  0ansson, 

1995) 

-  3  levels  of  training  1 )  causal  loop,  2)  strategic  time  plots,  and  3)  strategic 

heuristics)  (Maxwell,  1995) 
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Concuttent 
verbalization  / 
thinking-aloud 

-  Whether y  whik  piquing  the  game y  subjects  are  required  to  verbal^  describe  tasks 
and  heuristics  employed  (Berry  et  al.,  1984;  McGeorge  et  al.,  1989; 

Stanley  et  al.,  1989) 

Increasing  task 
salience 

-  Between  trial  blocks,  instruct  subjects  with  task  structures  and  effects  of  decisions 

and  time  delcy  (Berry  et  al.,  1988;  Wang,  1994) 

-  Whether  subjects  are  informed  with  appearance  of  delay  (Brehmer,  1995) 

Degree  of  dec’n 
precision  reqM 

-  Whether  subjects  are  required  to  place  decisions  to  the first  decimal  place 
(Sanderson,  1989) 

Learning 

inducement 

-  Prior  to  tasks,  instruct  subjects  to  focus  on  searchingfor  task  pattern  and  structure 

(Berry  et  al.,  1988) 

-  Prior  to  tasks,  induce  learning  iy  instructing  subjects  that  learning  is  crucial  and 

task  performance  does  not  affect  economic  reward  (Wang,  1994) 

Contents  of 
information 
display 

-  Whether  Bayesian  strategy  is  available  (Kleinmuntz  et  al.,  1987) 

-  Whether  subjects* premous  decisions  and  outcomes  are  available  (Sanderson,  1989) 

-  3  levels:  1 )  Feedforward:  whether  the  subjects  learned  the  three  formula;  2)  cognitive 

feedback:  whether  the  subjects  received  task  information;  3)  outcome  feedback: 
project  status  reports  in  numerical forms  (Sengupta  et  al.,  1993) 

Forms  of 

information 

display 

-  Whether  subjects  receive  graphical  representations  of  ystem  status  (McGeorge  et 

al.,  1 989;  Sanderson,  1 989) 

-  Whether  subjects  receive  formula  for  decisions  (Sanderson,  1989) 

-  Whether  subjects  only  receive  variables*  names  without  semantic  meanings 

(Sanderson,  1989) 

-  3  levels:  1)  no  cue  highlighted,  2)  all  cues  highlighted  (cue  discovery),  and  3)  all  cues 

highlighted plus  heuristics  (feedforward)  (Richardson  &  Rohrbaugh, 

1990)-  Improved  graphics  display  (Richardson  &  Rohrbaugh,  1990, 

Howie  et  al.,  2000) 

Decision-making 

architectures 

-  Whether  subjects  use  hierarchical  or  networked  decision-making  (Brehmer,  et  al., 
1995) 

APPENDIX  C 


Sterman  “Optimal”  Solution  in  Computer  Game 


FINAL  SCORE  =  19 

Press  <C><RTN>  to  continue. 

Graphic  taken  from  STRATEGEM-2  for  DOS  ©1985  by  John  Sterman 

In  year  four,  when  a  step  increase  in  goods  orders  rises  from  450  units  to  500,  the 
optimal  solution  produces  an  order  of  260  units  for  the  capital  sector.  In  the  following 
year,  zero  units  are  ordered.  For  year  eight,  10  units  are  ordered.  Year  10  orders  are  20, 
and  in  year  12,  60  capital  sector  orders  are  made  and  remain  that  way  for  the  remainder 
of  the  game  -  producing  the  optimal  score  of  19. 
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APPENDIX  D 


Original  Experiment  Instructions 

Welcome  to  the  STRATEGEM-2  Simulation  Game* 
Version  2.1  Copyright  1985  John  Sterman 


The  economic  malaise  of  the  1980’s  has  revived  interest  in  the  economic  long  wave 
or  Kondratiev  Cycle,  a  cycle  of  prosperity  and  depression  averaging  50  years. 

Since  1975  the  System  Dynamics  National  Model  has  provided  an  increasingly  rich 
theory  of  the  long  wave.  The  theory  emerging  from  the  National  Model  explains  the 
long  wave  as  the  endogenous  result  of  decision  making  by  individuals,  corporations,  and 
government.  However,  the  complexity  of  the  National  Model  makes  it  difficult  to 
explain  the  dynamics  underlying  the  long  wave.  This  game  demonstrates  how  long 
waves  can  arise  by  focusing  on  the  role  of  capital  investment. 

There  are  two  basic  kinds  of  industries  in  modem  economies:  capital  producers 
and  producers  of  consumer  goods  and  services.  Goods  producers  sell  primarily  to  the 
public.  Producers  of  capital  make  and  sell  the  plant  and  equipment  that  the  consumer 
sector  needs  in  order  to  produce  goods  and  services.  But,  in  addition,  the  capital- 
producing  industries  of  the  economy  (construction,  heavy  equipment,  steel,  mining, 
and  other  basic  industries)  supply  each  other  with  the  capital,  plant,  equipment,  and 
materials  each  need  to  operate.  Viewed  as  a  whole,  the  capital  sector  of  the  economy 
orders  and  acquires  capital  from  itself. 

You  will  manage  the  capital  producing  sector  of  the  economy.  Your  goal  is  to 
balance  the  supply  and  demand  for  the  capital.  To  do  this  you  must  keep  your 
production  capacity  (current  capacity)  as  closely  matched  to  the  demand  (total 
backlogs)  for  capital  as  possible.  The  game  is  won  by  the  person  with  the  lowest  score. 
The  score  is  the  average  absolute  deviation  between  production  capacity  and  desired 
production.  For  example,  if  capacity  were  500  and  demand  were  600,  your  score  for 
that  period  would  be  100.  Likewise,  if  capacity  were  600  and  demand  were  only  500, 
your  score  for  that  period  would  also  be  100.  A  Score  of  zero  means  supply  and 
demand  are  in  perfect  balance.  You  are  therefore  penalized  for  excess  capacity  (which 
implies  some  of  your  factories  are  idle)  and  also  for  insufficient  capacity  (which 
means  you  are  unable  to  meet  the  demand  for  capital). 

Time  is  divided  into  two-year  periods.  At  the  beginning  of  each  period,  orders  for 
capital  are  received  from  two  sources:  the  goods  sector  and  the  capital  sector  itself. 


*  These  instructions  were  taken  from  the  Howie  STRATEGEM-2  interface  (2000). 
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Orders  for  capital  arriving  from  the  goods  sectors  are  determined  by  the  computer. 
Orders  for  capital  you  placed  in  the  previous  period  are  moved  into  the  unfilled  order 
backlog  for  the  capital  sector. 

Orders  placed  by  the  goods  and  capital  sectors  accumulate  in  the  backlog  of 
unfilled  orders  for  each  sector.  The  total  backlog  of  orders  is  the  desired  production 
for  the  current  two-year  period,  the  demand  you  must  meet. 

Production  itself  is  the  lesser  of  desired  production  or  production  capacity. 
Production  capacity  is  determined  by  the  capital  stock  of  the  sector.  Capital  stock  is 
decreased  by  depreciation  and  increased  by  shipments.  You  lose  10%  of  your  stock 
each  period. 

If  capacity  is  inadequate  to  meet  demand  fully,  available  production  of  capital  is 
allocated  between  the  capital  and  goods  sectors  in  proportion  of  their  respective 
backlogs.  For  example,  if  the  backlog  firom  the  capital  sector  were  500  and  the 
backlog  from  the  goods  sector  were  1000,  desired  production  would  be  1500. 

If  capacity  were  only  1200,  production  would  be  1200  and  the  fraction  of  demand 
satisfied  would  be  1200/1500  =  80%.  Thus  400  units  would  be  shipped  to  the  capital 
sector  and  800  would  be  shipped  to  the  goods  sector. 

Any  unfilled  orders  remain  in  their  respective  backlogs  to  be  filled  in  future 
periods.  In  the  example,  100  units  would  remain  in  the  backlog  of  the  capital  sector 
and  200  would  remain  in  the  backlog  of  the  goods  sector. 
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APPENDIX  E 


Bois  Instructions 


This  section  will  be  an  on-screen  tutorial  provided  to  participants.  Below  are  pertinent 
views  of  the  tutorial. 


First  page:  Contains  navigation  instructions. 
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Game  board  overview:  Has  multiple  overlays  used  to  familiarize  participant  with  game  board 
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Fourth  view:  Explains  the  Kondratiev  Cycle 


The  Kondratiev  Cycle 


» 

4 

n 

1 

4 

1 

The  Kondratiev  Cycle,  or  long  wave,  is  characterized  by 
successive  waves  of  overexpansion  and  decline  of  the 
economy,  particularly  the  capital  producing  sectors. 
Overexpansion  means  an  Increase  in  the  capacity  to  produce 
factories,  equipment,  and  goods  relative  to  the  amount  needed 
to  replace  worn-out  units  and  provide  for  growth  over  the  long 
run.  Overexpansion  Is  undesirable  because  eventually, 
production  and  employment  must  be  cut  back  below  normal  to 
reduce  excess. 

To  illustrate,  consider  the  development  of  the  US  economy 
after  Worid  War  II.  The  capital  stock  of  the  economy  was  old 
and  severely  depleted  after  fifteen  years  of  depression  and  war. 
Demand  for  all  types  of  capital  -  factories,  machines,  roads, 
houses,  schools  -  surged.  A  massive  rebuilding  began.  In  order 
to  replace  its  worn-out  infrastructure,  the  capital  producing 
sector  had  to  expand  beyond  the  long-run  needs  of  the 
economy.  The  necessary,  Inevitable  overexpansion  of  the 
capital  sector  was  exacerbated  by  self-ordering.  As  the  demand 
for  consumer  goods,  services,  and  housing  rose, 
manufacturers  of  capital  plant  and  equipment  had  to  expand 
their  own  capacity,  further  swelling  demand.  This  self-ordering 
powered  the  boom  of  the  1950s  and  1960s.  By  the  late  1960s, 
however,  the  capital  stock  had  been  largely  rebuilt,  and 
investment  began  to  slow  to  a  level  consistent  with 
replacement  and  long-run  growth.  Excess  capacity  and 
unemployment  began  to  show  up  in  basic  industries.  Faced 
with  excess  capacity,  investment  was  cut  back,  further 
reducing  the  need  for  capital  and  reinforcing  the  economic 
decline  experienced  during  the  1970s. 


Fifth  view:  Explains  goal  /  scoring  of  simulation  with  links  to  other  explanations. 


Your  Mission 


As  the  manager  of  the  Bwaland  economy, 
it  is  your  goal  to  balance  the  supply  and 
demand  of  the  capital  sector.  To  do  so, 
you  must,  to  the  best  of  your  ability,  keep 
your  current  capacity  matched  to  the 
total  backlogs  of  all  orders. 

You  will  be  scored  on  how  well  you  are 
able  to  meet  your  goal,  A  score  of  zero 
means  that  current  capacity  and  total 
backlogs  (supply  and  demand)  are  in 
perfect  balance.  In  order  to  better 
understand  the  scoring  concept,  think  of 
this:  You  are  penalized  for  inefficient 
capacity  (which  implies  that  some  of 
your  factories  are  idle)  and  also  for 
insufficient  capacity  (which  means  that 
you  are  unable  to  meet  the  total  demand 
for  capital). 

The  bottom  line  on  scoring:  The  lower 
the  score,  the  better  you  are  performing! 
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Sixth  view:  Outlines  the  play  of  the  game. 


Game  Play 


During  the  game,  one  period  of  play  is  equal  to  two 
years.  You  will  begin  In  year  zero.  At  the  beginning  of 
each  period,  orders  for  capital  are  received  from  two 
sources:  the  goods  sector  (which  are  placed  by  the 
computer)  and  the  capital  sector  itself.  You  will  be 
making  the  capital  sector  order  inputs,  therefore  you 
must  keep  watch  on  how  many  goods  sector  orders 
are  being  made  at  the  same  time. 

Upon  clicking  the  order  button,  orders  for  both  the 
capital  and  goods  sectors  are  moved  into  their 
respective  backlog  portions  of  the  game  board  where 
they  accumulate.  As  you  know,  these  two  backlogs 
represent  the  total  backlog  of  orders  as  well  as  the 
demand  that  you  must  meet. 

Production  of  orders  cannot  be  greater  than  the 
current  capacity.  Additionally,  production  cannot  be 
greater  than  the  total  backlogs  (in  other  words, 
production  will  be  the  lesser  of  total  capacity  or  total 
backlogs).  Additionally,  the  capital  stock  (which 
represents  your  current  capacity),  is  depreciated  by 
10%  for  each  period  of  play.  This  Is  Important  to 
remember  when  placing  your  orders  for  capital  stock; 
Did  you  take  into  account  what  you  will  lose  to 
depreciation? 


Seventh  view:  Explains  how  order  allocations  are  made  to  each  sector. 


Allocation  of  Orders 


mm 


Production  allocation  is  as  follows: 


If  current  capacity  is  inadequate  to  meet  total 
backlogs  fully,  available  production  of  capital  is 
then  allocated  between  the  capital  and  goods 
sectors  in  proportion  of  their  respective 
backlogs.  For  example,  if  the  backlog  from  the 
capital  sector  were  500  and  the  backlog  from 
the  goods  sector  were  1000,  desired  production, 
or  total  backlogs,  would  be  1500. 

If  current  capacity  were  only  1200,  production 
would  be  1200  and  the  fraction  of  demand 
satisfied  would  be  1200/1500,  or  80%.  Thus  400 
units  would  be  shipped  to  the  capital  sector  and 
800  would  be  shipped  to  the  goods  sector. 

Any  unfilled  orders  remain  in  their  respective 
backlogs  to  be  filled  in  future  periods.  In  the 
example,  100  units  would  remain  in  the  backlog 
of  the  capital  sector  and  200  would  remain  in 
the  backlog  of  the  goods  sector. 

Remember;  This  allocation  process  creates 
delays  in  your  system  that  you  should  try  to 
anticipate. 
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Eighth  view:  Introduces  the  concept  behind  game  equilibrium. 


Important  Concept  to  Master:  Equilibrium 


♦  In  order  to  understand  the  STRATEGEM-2 
simulation  of  the  Kondratiev  Cycle,  it  is  critical  that 
you  understand  the  concept  of  equilibrium. 

•  The  equilibrium  level  is  the  current  goods  orders 
PLUS  depreciation. 

•  For  example,  if  current  capacity  were  650  and  total 
backlogs  were  also  650,  you  are  In  equilibrium.  At 
this  point  you  would  only  have  to  order  70  units  of 
capital.  This  is  because  capital  depreciation  would 
be  70  (actually,  the  10%  depreciation  is  65, 
however  the  game  rounds  to  the  nearest  10,  hence, 
an  order  for  70). 

*  When  in  equilibrium,  you  must  only  order  enough 
to  cover  the  depreciation  of  your  capital. 


Ninth  view:  Explains  how  one  manages  equilibrium. 

Includes  link  to  4-question  exam  used  to  bolster  learning  (not  shown) 


■ 


Important  Concept:  Managing  Equilibrium 
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Once  you  understand  the  concept  of 
equilibrium,  you  should  also  understand  that 
when  current  capacity  rises  above  the 
equilibrium  level,  it  will  drive  down  any  excess 
that  exists  in  the  total  backlogs  (this  is  good). 

Additionally,  when  current  capacity  is  below  the 
equilibrium  level,  it  will  drive  down  current 
capacity  and  cause  total  backlogs  to  increase. 

You  must  keep  an  eye  on  how  many  goods 
sector  orders  are  being  made  during  each 
period  of  play.  The  final  equilibrium  level  you 
will  be  shooting  for  will  be  equal  to  goods  sector 
orders  PLUS  the  depreciation  value  for  that  level 
of  orders  (goods  orders  times  10%).  When 
current  capacity  is  above  the  equilibrium  level, 
total  backlogs  will  decline.  And,  when  current 
capacity  is  below  the  equilibrium  level,  current 
capacity  will  decline  and  total  backlogs  will  go 
up. 
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Tenth  view:  Explains  the  sections  of  the  game  board  -  a  multiple  view  display. 


The  Game  Board  Explained 


■  B3 1  iBMf .TOl 


This  IS  the  part  that  you  control,  the  capital 
stock  order  placement  section.  You  click 
on  the  10s  and  100s  buttons  to  accumulate 

.  pave  ordered 

The  final  portion  of  the  game  board  that  you  may  find  negative 
helpful  are  these  five  graphs  (there  is  no  graphical  op^er.  Once 
information  shown  at  this  time  as  this  game  board  is  )rder'  button 
in  year  zero).  As  each  period  of  play  goes  by,  you  ^  gg^g 
can  refer  to  these  graphs  to  show  you  historical  data  ^^gg  gf 
of  your  game  play.  This  may  or  may  not  be  helpful  to  ^  window  to 
you,  however,  it  is  provided  in  order  to  display  (j^g  buttons, 
feedback  information  about  your  decisions.  “Order," 

there  will  be  no  turning  back  i  you  made  a 
e.  So  please  be  careful. 


This  completes  the  game  board  portion.  If  you  need  to  reread  a 
previous  display  of  the  game  board,  simply  right  click  until  you  get  to 
the  specific  portion  you  are  looking  for.  Otherwise,  click  to  continue. 


Final  view:  Provides  tips  to  remember. 


Don’t  Forget 


The  equilibrium  level  is  determined  by 
using  the  current  goods  sector  orders  plus 
depreciation. 

To  decrease  the  total  backlogs,  current 
capacity  must  be  above  the  equilibrium 
level. 

To  increase  current  capacity,  capital 
orders  must  exceed  the  amount  of  current 
depreciation. 

To  decrease  the  current  capacity,  capital 
orders  must  be  below  the  amount  of 
current  depreciation. 

Decisions  you  make  in  a  specific  period  of 
play  will  not  show  up  for  at  least  2  to  4 
more  years. 

Please  do  not  forget  that  you  are  trying  to 
balance  current  capacity  with  the  total 
backlogs  at  the  equilibrium  level. 


■iw 
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First  linked  view;  Accessed  only  from  another  page.  Used  to  define  Bwaland. 


Second  linked  view:  Used  to  further  define  the  capital  and  goods  sectors. 


“Capital’  and  “Goods”  Sectors  of  Bwaland 


Think  of  the  Capital  Sector  as  that  portion 
of  your  economy  that  represents  all  of 
your  industry.  It  may  include  such  things 
as  power  generation,  water  resources, 
mining,  fuel  production,  agriculture, 
textiles,  heavy  and  light  manufacturing, 
and  factories  that  produce  “goods”  for 
consumers,  as  well  as  parts 
manufacturing  and  the  production  of 
equipment  for  Itself. 


The  Goods  Sector  is  that  portion  of  your 
economy  that  actually  consumes  what  is 
produced,  it  may  include  such  items  as 
electricity,  water,  gasoline,  heating  fuels 
and  gases,  food,  clothing,  and  all  items 
that  can  be  found  on  customer  shelves  In 
various  stores  and  outlets. 
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Third  linked  view:  Used  to  further  define  current  capacity  and  total  backlogs 


I*  The  Current  Capacity  is  equal  to  your  total 
Capital  Stock.  It  indicates  how  much  you  can 
produce  to  satisfy  the  needs  of  the  goods 
sector  and  your  own  capital  sector.  What  is 
important  remember  here  is  that  you  will  lose 
10%  of  your  capital  stock  each  period  of  play 
due  to  depreciation.  Therefore,  you  must 
always  consider  that  when  you  place  an  order, 
are  you  also  including  enough  to  cover 
expected  losses  due  to  depreciation. 


Total  Backlogs  -  Demand.  This  number  is  the 
sum  of  all  goods  sector  orders  and  backlogs 
combined  with  all  capital  sector  orders  and 
backlogs.  Because  you  are  dealing  with  a  time 
delay,  the  total  backlogs  reflects  decisions  that 
were  made  two  years  ago  (one  period  of  play). 
To  be  an  effective  player,  this  means  that  you 
must  anticipate  what  this  level  will  be  one 
game  period  ahead  of  time.  In  other  words, 
when  facing  a  given  game  screen  for  a 
particular  period  of  play,  it  will  behoove  you  to 
remember  what  you  have  ordered  in  the  past. 


Fourth  linked  view:  Explains  how  the  game  is  scored. 


APPENDIX  F 


The  Howie  STRATEGEM-2  Interface 

The  following  depiction  of  the  STRATEGEM-2  interface  shows  the  beginning  of  the 
game.  It  indicates  a  goods  sector  demand  of  450  orders  with  an  overall  capacity  of  500. 
The  participant  would  need  to  only  order  50  units  in  the  capital  sector  (which  is  just 
enough  to  accommodate  depreciation).  The  50  capital  orders  combined  with  the  450 
goods  sector  orders  equals  the  500  units  of  total  capacity  and  would  therefore  keep  the 
game  in  equilibrium  and  keep  the  score  at  zero. 


Capital  Stock  Order  Placement 

1000  500  0 


Time 


Goods  Sector  Demand 


70 


^08  I  “ID  I  Of^er  ]^18  I  ♦ISO 


450 


Set  your  new  Caintal  Sector  Order 
osinrj  the  ot  ente?  a 

value  here:^ 


10DO  SOD 


SCORE 

0 


50 


Dial 


Current  Capacity:5d0 


lODO 

CLirrent  Depreciation:  50 


'  1.  willing  for  Capital  Sector  Order 
2.  Shipping  Capital  Sector  Goods 
3..  Shipping  Goods  Sector  Detrvcncs 

4.  Oeprecialing  Capital  Stock 

5.  Adding  New  Demands 
E-  Updating  Backlogs 

7,  Adding  Shipped  Caphal  Goods 


450 


1000 


New  Orders  Goods  Desfred  if^oijut^on 


Capital 


Production 


New  Orders  Capilat 
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The  depiction  shown  below  is  in  year  32  of  a  sample  game  played  by  the  researcher.  At 
this  pomt,  the  researcher  has  allowed  the  current  capacity  (520)  to  be  too  low  in  order  to 
meet  the  total  demand,  or  desired  production  (500)  plus  accommodate  depreciation  (50). 
Therefore,  current  capacity  should  at  least  be  550  to  maintain  the  game  in  equilibrium  at 
this  point.  The  depiction  below  occurred  from  under-ordering  in  the  previous  timeframe. 

In  response,  the  researcher  is  ordering  90  capital  units  in  order  to  boost  capacity  in 
future  years. 


Cap^a!  Stock  Order  Placement 

1CCC  50Q  ^ 


Time 


1 

■IW  1  -10  1  Ortcr 

tl8  j  ♦IBOJ 

90 


Goods  Sector  Demand 


500 


Sector  OfMer 
using  butloas  above  or  enter  a 
nowVdlue 


SCORE 

153 


►  t.  Waiting  for  Capital  Sector  Order 

2.  Shipping  Capital  Sector  Goods 

3.  Shipping  Goods  Sei:tor  DeUveries 

4.  Depredating  Capital  Stack 
S«  Adding  New  Demands 

8.  Updating  Gaddogs 
r.  Adding  Shipped  Capital  Goods 


Current  CBpacity;520 


1000 


s 

^  m 

c 

r  0 

0 

R 

1®  500  ' 

- -I _  ... 

J  £ 

1  500  1000  C 

1  Total  Bac 

m 

500  ° 

10D0 

Current  Depredation:  60 

11 

- 

1  New  Orders  Goods  Ocstred  f^oductl’on 

Capital 

Production  h 

few  Orders  Capital 
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This  final  depection  shows  the  sample  game  in  the  final  year  (year  70).  The  researcher 
has  managed  to  get  the  game  back  in  equilibrium  (this  occurred  in  year  68).  At  this 
point,  orders  required  for  the  capital  sector  need  only  to  accommodate  the  current 
depreciation  (60  units).  The  final  score  for  the  game  is  167. 


|#STATA&‘’t 


mmm 


Capital  Stock  Order  Placement 


Goods  Sector  Demand 


500 

Q 

D 

TO  0 

60 

■ 

■1 

Se<  your  new  Capital  Sector  Order 
new  value  here: 


SCORE 
167  . 


1.  Waiting  ior  CapHat  Sector  Order 
Z,  Shipping  Capital  Sector  Goods 

3.  Shipping  Goods  Sector  Deliveries 

4.  OepredatSfig  Capital  Slock 

5.  Adding  New  Demands 

6.  Updating  Backlogs 

7.  Adding  Shipped  Capital  Goods 


Current  Cpipacify:5SP 


1000 

Current  Depreciation:  60 


New  Orders  Goods  Desired  Production  Capital 


Production  New  Orders  Capital 


Knowledge  Survey 

Adapted  from:  Howie,  E.,  Sy,  S.,  Ford,  L.,  &  Vicente,  K.  J.  (2000).  Human-computer 
interface  design  can  reduce  misperceptions  of  feedback.  System  Dynamics  Review,  16(^),  151- 
171. 

Participant  Number _ 

1 .  Is  there  depreciation  on  “goods”  shipped  to  the  goods  sector? 

a)  Yes 

b)  No 

2.  What  is(are)  the  main  sector(s)  in  the  economy? 

a)  Depreciation 

b)  Goods 

c)  Capital 

d)  A  and  C  only 

e)  B  and  C  only 

f)  A  and  B  only 

3.  Of  which  sectors  (use  question  2’s  options)  do  you  have  control  over? 

a)  Depreeiation 

b)  Goods 

c)  Capital 

d)  A  and  C  only 

e)  B  and  C  only 

f)  A  and  B  only 
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4.  If  the  current  capacity  increases  and  any  other  demands  stay  the  same,  the 
subsequent  total  backlog: 

a)  Increases 

b)  Decreases 

c)  Does  not  change 

d)  The  current  capacity  is  irrelevant  to  the  backlog 

5.  The  Kondratiev  long  wave  is  used  to  depict  the  results  of  over-expansion  in  a  capital 
sector: 

a)  True 

b)  False 

6.  Total  Backlog  consists  of: 

a)  Capital  sector  orders 

b)  Capital  depreciation 

c)  Goods  sector  orders 

d)  All  of  the  above 

e)  A  and  C  only 

f)  None  of  the  above 

7.  The  factor(s)  that  contributes  to  the  game  score  is(are): 

a)  Current  Capacity 

b)  Total  Backlogs 

c)  The  Game  Period 

d)  All  of  the  above 

e)  A  and  B  only 
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8.  To  get  the  best  score,  you  should: 

a)  Maximize  overproduction 

b)  Minimize  over  and  underproduction 

c)  Maximize  underproduction 

d)  None  of  the  above 

9.  How  does  the  current  capacity  increase? 

a)  Capital  sector  shipments 

b)  Goods  sector  shipments 

c)  Depreciation 

d)  None  of  the  above 

10.  How  does  current  capacity  decrease? 

a)  Capital  sector  shipments 

b)  Goods  sector  shipments 

c)  Depreciation 

d)  None  of  the  above 

1 1 .  Depreciation  consists  of: 

a)  The  consumption  of  capital  goods  by  the  goods  sector 

b)  Lost  orders  in  transit  to  the  goods  sector 

c)  Capital  orders  produced  for  the  capital  sector’s  consumption 

d)  Reduction  in  current  capacity  from  wear  and  tear 

12)  Does  depreciation  reduce  the  capacity  of  the  capital  sector? 

a)  Yes 

b)  No 

c)  Not  applicable 


129 


13.  The  capital  sector: 

a)  Produces  goods  to  be  consumed  by  the  capital  sector 

b)  Produces  goods  to  be  consumed  by  the  goods  sector 

c)  Consumes  the  depreciated  material 
d  )  All  of  the  above 

e)  A  and  B  only 

14.  Goods  leave  the  production  system  via: 

a)  Shipment  to  the  goods  sector 

b)  Shipment  to  the  capital  sector 

c)  Depreciation 

d)  A  and  C  only 

15.  If  the  capital  sector  is  running  at  full  capacity,  and  the  goods  sector  demand 
increases,  in  the  next  period  of  play,  meeting  the  increase  in  demand  will  cause: 

a)  The  capital  sector  demand  to  increase 

b)  Underproduction 

c)  A  backlog  in  the  goods  sector  shipments 

d)  All  of  the  above 

e)  None  of  the  above 

16.  If  the  current  capacity  is  larger  than  the  total  backlogs,  what  will  occur  to  the  game 
score? 

a)  It  will  go  up 

b)  It  will  go  down 

c)  It  will  remain  the  same 


Continue  on  to  next  page 
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In  this  section,  match  an  answer  to  the  following  questions.  Note  there  are  more 
answers  than  there  are  questions.  You  may  refer  to  the  attached  diagram  of  the  game 
board  to  assist  you  with  some  of  the  questions. 


Questions: 

1 .  What  is  represented  by  the  field: 
Capital  Stock  Order  Placement? 

Answer: _ 

2.  What  is  represented  by  the  field: 
Goods  Sector  Demand? 

Answer: _ 

3.  How  do  you  obtain  desired 
production? 

Answer: _ 

4.  How  does  the  simulation  calculate 
the  current  capacity? 

Answer: _ 

5.  What  is  represented  by  the  current 
capacity  field? 

Answer: _ 

6.  What  factors  are  involved  in 
calculating  the  allocation  of  orders? 

Answer: _ 

7.  What  factors  are  involved  in 
computing  your  score? 

Answer: 


Answers: 

A)  Add  goods  sector  capacity  plus 
capital  sector  capacity 

B)  Ratio  of  capital  sector  backlog  to 
goods  sector  backlog 

C)  Add  capital  sector  backlog  plus 
goods  sector  backlog  (total 
backlogs) 

D)  Cumulative  total  of  previous  year’s 
over/under  production  divided  by 
the  game  period 

E)  Ratio  of  goods  sector  backlogs  to 
depreciation 

F)  Goods  order  placements 

G)  Current  capacity  and  total  backlog 

H)  Capital  sector  demand 

I)  Total  production  capability 

J)  Depreciate  capital  stock,  then  add 
capital  sector  shipments  to  current 
capacity 

K)  Ratio  of  current  capacity  to 
depreciation 


8.  How  is  the  current  capacity 
proportioned? 

Answer: 
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Answers  to  Knowledge  Survey 

(not  provided  to  participants) 


Multiple- 

-Choice  Questions 

Matching  Questions 

l.B 

9.  A 

l.H 

2.E 

10.  C 

2.F 

3.C 

11. D 

3.C 

4.B 

12.  A 

4.  J 

5.  A 

IS.BorE 

5.1 

6.E 

14.  D 

6.G 

7.D 

15.  D 

7.D 

8.B 

16.  A 

8.B 
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APPENDIX  H 


Self-Assessment  Survey  Cover-Sheet 

Decisions  Within  Complex  Systems:  An  Experimental 
Approach  Using  the  STATEGEM-2  Computer  Game 

Researcher:  J.  Robert  Bois 

This  survey  has  been  approved  by  the  Institutional  Review  Board 
State  University  of  New  York,  Albany 


Dear  Participant, 

Y ou  have  just  taken  part  in  a  voluntary  experimental  study  of  decision  making  within  a 
dynamic  system.  Your  personal  results  will  always  be  kept  totally  confidential  (known 
only  by  you  and  the  researcher).  The  combined  results  of  this  study  will  assist  other 
researchers,  as  well  as  decision  makers  in  complex  environments,  to  better  understand 
the  nature  of  the  dynamic  decision-making  process.  At  this  time,  you  are  being  asked  to 
participate  in  a  voluntary  written  survey.  It  is  designed  to  assess  the  experiment  and  to 
gather  feedback  from  you  on  your  perceptions  as  a  participant.  The  answers  you  provide 
here  are  expected  to  only  enhance  the  research  findings.  The  same  rule  of  confidentiality 
expressed  as  being  part  of  the  experiment  also  applies  to  this  survey. 

Additionally,  you  are  reminded  that  you  are  participating  freely  and  that  you  are  under 
no  obligation  to  answer  any  or  all  of  the  questions.  This  survey  should  take,  on  average, 
no  more  than  five  minutes  to  complete.  Thank  you  for  your  participation. 

Please  turn  the  page  and  be  sure  to  read  all  instructions  before  beginning. 
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Self-Assessment  Survey 


Participant  Number 


Please  consider  the  following  statements  carefully.  After  each  statement,  circle  the  answer  that  best 
reflects  your  opinion.  Would  you  say  you  strongly  agree  with  the  statement,  agree,  are  neutral,  disagree,  or 
strongly  disagree?  Mark  your  answers  accordingly  on  the  scale  for  each  question.  As  a  reminder,  you 
should  answer  each  question  as  truthfully  as  possible.  There  are  no  wrong  answers  unless  you  are  not 
being  completely  honest  with  yourself.  Please  go  to  the  first  question  and  begin. 


Circle  your  response  to  each  question. 

1 .  Regarding  this  survey,  I  fully  understand  all  that  is  required  of  me  from  the  instructions. 


1  2  3  4  5 

Strongly  Strongly 

Disagree  Disagree  Neutral  Agree  Agree 

2.  Regarding  the  experiment,  I  fully  understood  all  that  was  required  of  me  from  the  instructions. 


12  3  4 

Strongly 

Disagree  Disagree  Neutral  Agree 

3. 1  did  my  best  in  performing  during  the  experiment. 

12  3  4 

Strongly 

Disagree  Disagree  Neutral  Agree 

4.  The  experiment  took  too  much  time  to  complete. 

12  3  4 

Strongly 

Disagree  Disagree  Neutral  Agree 

5.  During  the  experiment,  I  sometimes  forgot  what  I  was  supposed  to  do. 

12  3  4 

Strongly 

Disagree  Disagree  Neutral  Agree 

6.  Time  constraints/pressures  made  me  hurry  during  my  responses. 

12  3  4 

Strongly 

Disagree  Disagree  Neutral  Agree 


5 

Strongly 

Agree 


5 

Strongly 

Agree 


5 

Strongly 

Agree 


5 

Strongly 

Agree 


5 

Strongly 

Agree 
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7.  When  provided  with  a  set  of  decision  cues  to  follow,  I  followed  them  all  the  time. 


1 

2 

3 

4 

5 

strongly 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Agree 

8. 1  found  that  the  knowledge  survey  was  very  difficult  to  accomplish 

1 

2 

3 

4 

5 

Strongly 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Agree 

9.  The  required  tasks  were  easy  to  understand. 

1 

2 

3 

4 

5 

Strongly 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Agree 

10.  There  were  times  when  I  found  myself  bored  with  completing  the  tasks. 

1 

2 

3 

4 

5 

Strongly 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Agree 

11.1am  very  interested  in  the  outcomes  of  this  research  project. 

1 

2 

3 

4 

5 

Strongly 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Agree 

12.  1  gave  this  experiment  “my  all.” 

'  1  performed  exactly  in  the  manner  prescribed  in  the  verbal 

and  written  instructions. 

1 

2 

3 

4 

5 

Strongly 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Agree 
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Self-Assessment  Survey  Request  for  Comments 


In  the  space  provided  below,  please  write  any  comments  you  would  like  to  have  known 
to  the  researcher.  If  you  need  more  space,  please  use  the  reverse  side  of  this  page. 
Additionally,  if  in  the  future  you  would  like  to  get  in  touch  with  the  researcher;  his 
contact  information  is  provided  at  the  bottom  of  this  form. 


You  have  just  taken  part  in  a  voluntary  written  survey  regarding  your  participation  in  an 
experimental  study  of  decision  making  within  a  dynamic  system.  I  would  like  to  remind 
you  that  your  answers  and  comments  in  this  survey  are  to  be  kept  totally  confidential  by 
the  researcher.  As  an  added  note,  you  are  also  asked  to  not  discuss  this  survey  with  any 
other  participant  until  after  all  data  collection  is  complete.  If  you  have  any  further 
questions,  please  contact  the  researcher. 


A  sincere  thank  you  for  taking  the  time  to  participate  in  this  study. 

Researcher: 

J.  Robert  Bois 

bois  j  @nycap.rr.com 

(518)  877-8781 
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Appendix  I 

Variables  Collected  and  SPSS  Column  Codes 

1.  partcpnt  :  Participant  Number 

2.  group  :  Group  Number 

3.  gend  :  Gender  :  1  =  male;  2  =  female 

4.  age  :  Age  in  years 

5.  grad  :  SUNY  Status  :  1  =  undergraduate  student;  2  =  graduate  student 

6.  yr:Year  :  1  =  junior;  2=^  senior;  3=  graduate 

7.  prog  :  Program  or  Major  :  1  =  INFPhD;  2  =  PADPhD;  3  =  MPA;  4  =  MPP 

5  =  Marketing;  6  =  Business  Administration;  7  =  Other 

8.  exp  :  Years  professional  experienee 

9.  time  :  Timeslot  :  l=9am;  2=  12pm;  3  =  3pm;  4  =  6pm 

10.  tut  t  :  Time  on  tutorial 

11.  prae  t  :  Time  on  practiee 

12.  test_t  :  Time  on  test 

13.  garne  t  :  Time  on  game 

14.  tt  :  Total  time  for  experiment 

15.  tl  :  Trial  1  score 

16.  LoglOTl  :  Base  10  Logarithmic  conversion  of  Tl  score 

17.  t2  :  Trial  2  score 

18.  LoglOT2  :  Base  10  Logarithmic  conversion  of  T2  score 

19.  ta  :  Trail  average 

20.  LoglOTA  :  Base  10  Logarithmic  conversion  of  T A  score 

21 .  Delta  :  Change  in  score  by  subtracting  Logl0T2  from  LoglOTl 

22.  ts  :  Test  score 

23.  ql  :  Test  question  1  (0  =  incorrect  response;  1  =  correct  response) 

24.  q2  :  Test  question  2 

25.  q3  :  Test  question  3 

26.  q4  :  Test  question  4 

27.  q5  :  Test  question  5 

28.  q6  :  Test  question  6 

29.  q8  :  Test  question  8 

30.  q9  :  Test  question  9 

31.  qlO  :  Test  question  10 

32.  qll  :  Test  question  11 
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33. 

34. 

35. 

36. 

37. 

38. 

39. 

40. 

41. 

42. 

43. 

44. 

45. 

46. 

47. 

48. 

49. 

50. 

51. 

52. 

53. 

54. 

55. 

56. 

57. 

58. 


ql2  :  Test  question  12 
ql3  :  Test  question  13 
ql4  :  Test  question  14 
ql5  :  Test  question  15 
ql6  :  Test  question  16 

ml  :  Matching  question  1  (0  =  incorrect  response;  1  =  correct  response) 

ni2  :  Matching  question  2 

m3  :  Matching  question  3 

m4  :  Matching  question  4 

m5  :  Matching  question  5 

m6  :  Matching  question  6 

m7  :  Matching  question  7 

m8  :  Matching  question  8 

sal  :  Self-assessment  survey  question  1 

sa2  :  Self-assessment  survey  question  2 

sa3  :  Self-assessment  survey  question  3 

sa4  :  Self-assessment  survey  question  4 

sa5  :  Self-assessment  survey  question  5 

sa6  :  Self-assessment  survey  question  6 

sa7  :  Self-assessment  survey  question  7 

sa8  :  Self-assessment  survey  question  8 

sa9  ;  Self-assessment  survey  question  9 

salO  :  Self-assessment  survey  question  10 

sal  1  :  Self-assessment  survey  question  1 1 

sal 2  :  Self-assessment  survey  question  12 

sacom  :  Self-assessment  survey  comment  :  0  =  No,  l=Yes 
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APPENDIX  J 


Improved  Richardson  and  Rohrbaugh  Rule  Card 


As  the  manager  of  the  STRATEGEM-2  economy,  you  have  taken  it  upon  yourself  to  hire 
a  very  reputable  economic  consultant  to  assist  you  with  your  decisions.  This  person  has 
determined  that  if  you  are  to  follow  the  formula  in  the  box  on  the  reverse  side  of  this 
card,  you  will  most  likely  receive  an  outstanding  score  for  the  game.  You  are  reminded 
by  this  professional  that  although  you  are  not  required  to  heed  the  advice  given,  you  must 
remain  patient  and  diligent  with  using  the  formula  {use  the  reverse  side  for  success!) 

Example  on  using  the  decision  aide  in  year  zero  of  the  game: 

1 .  Take  the  current  depreciation  of  50  units  and  multiply  it  times  2  (for  1 00). 

2.  Add  to  that  the  shortfall*  (currently  0)  and  divide  by  2  (which  equals  0). 

3.  Then  subtract  the  current  capital  backlog  {not  total  backlog)  of  50. 

4.  This  produces  an  order  of  50  capital  units  for  Year  0. 

-  When  orders  end  in  a  “5,”  round  UP  to  the  nearest  10. 

-  If  orders  compute  to  less  than  zero,  use  zero  as  your  order. 

*  Shortfall  =  (total  backlogs  *  current  capacity)  -  always  use  the  true  value  of  this  number, 
(either  positive  or  negative).  For  example:  If  this  number  computes  to  less  than  zero,  then 
:  use  that  negative  number  and  therefore,  subtract  it  from  Depreciation. 


1.  Plan  in  advance  to  replace  depreciation  loss 

(DEPRECIATION  X  2)  =  _ 

2.  Shortfall:  Reconcile  total  backlogs  with  current  capacity 

add  or  subtract  (SHORTFALL*  -  2  )  =  (+/-) _ 

3.  Adjust  for  prior  orders  not  yet  filled 

subtract  (CAPITAL  BACKLOG  not  Total  Backlog)  =  - _ 

Total  Orders  =  _ 

(When  Total  Orders  compute  to  a  value  of  less  than  zero,  use  zero) 


*  Shortfall  =  (total  backlogs  -  current  capacity)  -  always  use  the  true  value  of  this 
number,  (either  positive  or  negative).  For  example:  If  this  number  computes  to 
less  than  zero,  then  use  that  negative  number  and  therefore,  subtract  it  from 
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